Colonoscopy system and method

ABSTRACT

Described are colonoscopy systems and methods of using such systems. The colonoscopy systems may include an optical scanning system having at least one illuminator configured to produce spatially patterned light and solid light in at least one frame to illuminate tissue within the colon, and at least one camera configured to capture the at least one image of the illuminated tissue within the colon. Additionally, the optical scanning system may include at least one control system configured to construct at least one three dimensional point cloud representations of the tissue within the colon.

INCORPORATION BY REFERENCE

The present patent application claims priority and benefit ofProvisional U.S. Ser. No. 62/951,094, filed Dec. 20, 2019; which claimspriority and benefit of Provisional U.S. Ser. No. 62/852,149, filed May23, 2019, which claims priority and benefit of Provisional U.S. Ser. No.62/810,784, filed Feb. 26, 2019; the entire contents of all of which arehereby incorporated herein by reference.

BACKGROUND

Colonoscopy provides a gold standard screening tool in the battle todetect and prevent colorectal cancer (CRC), the second leadingcancer-related killer in the U.S.; yet, even trained gastroenterologistsregularly miss polyps and adenomas that may lead to colorectal cancer. Astudy of back-to-back colonoscopy procedures found miss rates of 16.8%for polyps and 17.0% for adenomas, increasing to 20.7% and 22.9% forsmall (<6 mm) polyps and adenomas respectively, even when performed byexperienced personnel [1,2]. Including all procedures, anywhere from 3%to 5.5% of missed polyps become cancerous growths [1,3], and 6% ofpatients develop CRC within three years of a clear colonoscopy. Ashortage of endoscopy specialists and an ever-increasing over-50population compound the problem. In 2010, gastroenterologists performedjust about 50% of colonoscopies [4], and studies clearly show thatinexperienced practitioners are five times more likely to miss polypsand adenomas than experienced practitioners [3, 5]. In 2013, only 58% ofadults 50-75 remained up to date on screenings for CRC [6], driven inpart by lack of access to experienced endoscopic practitioners. Missedpolyps, adenomas, and cancers have direct consequences in both lives andmedical costs. Nearly 150,000 people are diagnosed with colorectalcancer every year, and 35% of CRC patients die from the condition [7].In 2010, direct medical costs for CRC care exceeded $14 billion [8], andprojected costs for 2020 may exceed $17 billion [9]. Detecting CRC atstage I instead of stage IV saves lives and reduces treatment costs by27% [9,10]. Reducing the gap between needed screenings and availableprofessionals would require over 7300 additional trained specialists[4], a professional population which current and projected medicalschool graduation rates cannot possibly meet. A safe, accurate,efficient, and augmented tool readily installed on the endoscopicinstrument and easily employed by both specialists and non-specialistsmay significantly improve accuracy and success rates of colonoscopy,increase efficiency to reduce procedure times, and improve patientaccess to screenings, leading to earlier detection and more effectivetreatment of CRC.

Existing colonoscopy systems utilize technology developed over ten yearsago that limits operator effectiveness in detecting abnormal tissue.Standard endoscopes utilize a visible light source and camera to viewthe colon. The operator can insert additional instruments through theendoscope tip to perform a polypectomy or collect samples frompotentially cancerous tissues. Ideally, current colonoscopy techniquescan find precancerous polyps and adenomas, facilitate removal ortreatment, and provide early detection of CRC. The CDC estimates thatcolonoscopies prevented 66,000 colorectal cancers between 2003 and 2007alone [8,10]. Despite the endoscope's capabilities and operatortraining, operators still miss polyps and adenomas that can lead tointerval cancers. Missed polyps and adenomas occur for several reasons.Polyps and adenomas, particularly ones under 6 mm in size, can grow infolds in the colon wall that block the operator's view. Abnormal tissuecan have similar coloring to that of surrounding tissue, reducing visualcontrast, causing abnormal tissue to blend into the background. The missrate increases as the number and density of polyps increases. Othercauses include poor bowel preparation, failure to examine the colonthrough to the cecum, and executing the procedure too quickly, which canresult in poor positioning of the camera, resulting in sections of thecolon going unscreened. Improving the efficacy of colonoscopy procedurestherefore requires development of new technologies and non-traditionalsystems that provide better tissue scanning and physician guidance.

Several proposed solutions within the prior art, such as narrowbandimaging, autofluorescence, virtual colonoscopy, and Third-Eye®retroscope [11-14] have proceeded as far as clinical trials. In all ofthese methods, clinical trials found the increase in adenoma detectionwas statistically insignificant and in some cases nonexistent, and whatsignificant difference did exist in one trial or set of patients couldnot be consistently maintained over successive trials. Several companiesdeveloped, tested, and commercialized systems that attempted to improvethe efficacy of colonoscopy procedures. The Endocuff, G-EYE, andEndoRings solutions employ mechanical systems that protrude from theendoscope's tip to flatten colon folds that may hide polyps and adenomas[15-17]. In interviews, physicians reported that the protrusions madeinserting the endoscope into the patient difficult, complicatedendoscope movement, increased procedure length, and increased potentialto injure or perforate the colon [14]. The Third Eye Panoramic Systememploys a module containing two side facing, wide angle source-camerapairs mounted on the endoscope's side [18]. The system displays threeseparate images on the screen, one per camera, requiring the operator tosimultaneously monitor and process three images to detect polyps insteadof a single, integrated view. The viewing difficulty, and lack ofdetection, location, and treatment assistance, does not significantlyincrease polyp detection rate. Magnetiq Eye employs deep-learningartificial intelligence (AI) techniques to examine 2D video endoscopeimages and identify suspicious tissue based on extensive training. Theeffectiveness of detecting missed adenomas depends strongly on thequality and extent of AI training data. Such methods have beendemonstrated to perform poorly, even when well trained, when working inan environment such as the colon that produces featureless twodimensional images, limiting effective reduction in miss rates. Theartificial intelligence also produced a very high rate of falsepositives for detected polyps using only two dimensional images forinput.

A commercial system exists that employs three dimensional magneticendoscopic imaging, using external sensors to track a magnetic marker onthe endoscope, to help the operator locate the endoscope within thepatient's colon with millimeter accuracy. Tracking endoscope positionallows identification and real-time treatment of colon loops [19,20].However the system cannot detect polyps and adenomas, cannot providenavigation guidance or other operator assistance, and does notminiaturize well. The limitations of proposed and commercially availabletechnologies highlight the need for an approach that integratesseamlessly with the endoscope to minimize hindrance to the surgicalprocedure, provides accurate and easy-to-use information to theperforming physician, and demonstrates improved detection and treatmentof polyps, adenomas, and cancerous tissue.

Optical scanning systems offer a candidate technology for producinghigh-accuracy three dimensional imagery and modeling of the colon.Current commercially available three dimensional optical scannersutilize a combination of technologies, including near-infrared (NIR) andvisible light, digital light projection (DLP) sources and verticalcavity surface emitting laser (VCSEL) projectors, and high-resolutioncameras, to produce three dimensional scans of an object. However, allcurrently available scanners are designed to operate in an environmentthat does not restrict the distance between scanner and object. In suchan environment, the scanner typically uses a large 8 cm) baseline—thedistance between centers of the optical source and/or recordingcameras—to achieve high resolution depth measurements [26-33]. Theenvironment does not restrict source and/or camera size, and thus thesystems use larger (dimensions up to 1 cm) components, particularly forthe cameras, to increase field of view and depth accuracy. However,scanning and measurement operations in the human body severely limitdistance between scanner and object and severely restrict scanningsystem size. In the colon, 5 cm at most separates a wall from center,which precludes the use of long baselines and large components. Systemswith larger dimensions and components would not integrate well with theendoscope, most likely resulting in protrusions or separate systems thatwould hinder the procedure in much the same manner as mechanicalsolutions.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS

To assist those of ordinary skill in the relevant art in making andusing the subject matter hereof, reference is made to the appendeddrawings, which are not intended to be drawn to scale, and in which likereference numerals are intended to refer to similar elements forconsistency. For purposes of clarity, not every component may be labeledin every drawing.

FIG. 1 is a block diagram of an exemplary colonoscopy system having anoptical scanning system, an AHRS unit, and a control system inaccordance with the present disclosure.

FIGS. 2A and 2B are perspective views exemplary endoscopic systemshaving exemplary optical scanning systems mounted on an outer wall.

FIG. 3 is a perspective view of an exemplary endoscopic system having anexemplary optical scanning system integrated within hardware of theendoscopic system.

FIG. 4A is a block diagram of an exemplary optical imaging systemcollecting light from an edge-emitting laser (EEL), and an opticallybased pattern generator having a diffractive optical element positionedafter the optical imaging system.

FIG. 4B is a block diagram of an exemplary optical imaging systemcollecting light a vertical cavity surface emitting laser (VCSEL), andan optically based pattern generator having a diffractive opticalelement positioned after the optical imaging system.

FIGS. 5A-5C are perspective views of exemplary projected patterns fromoptical sources without inclusion of diffractive optical elements.

FIGS. 6A-6C illustrate exemplary composite frames provided by anoptically-based pattern generator in accordance with the presentdisclosure.

FIG. 7A is a perspective view of an exemplary embodiment of an opticalscanning system having a parallel configuration of an optical source anda camera.

FIG. 7B is a block diagram of an exemplary embodiment of a colonoscopysystem having the optical scanning system of FIG. 7A.

FIGS. 8A and 8B are block diagrams of exemplary optical scanning systemshaving converged configurations of an optical source and camera.

FIGS. 9A-9C are block diagrams of exemplary optical scanning systemshaving converged configurations resulting in different overlap areas.

FIG. 10 is a block diagram of an exemplary colonoscopy system having atleast one camera and at least one structure optical source in accordancewith the present disclosure.

FIGS. 11-14, 15A and 15B illustrate exemplary embodiments of colonoscopysystems having at least one optical source and at least two cameraslocated at two different positions with respect to the optical source.

FIGS. 16A-16C are graphs illustrating dependence of field of view, deptherror and lateral error on depth between cameras and tissue wall of acolon.

FIG. 17 is a block diagram of an exemplary colonoscopy system having twooptical sources and at least one camera.

FIGS. 18 and 19 illustrate exemplary embodiments of colonoscopy systemshaving at least one optical source, at least one infrared camera and atleast one RGB camera.

FIGS. 20A and 20B illustrate a three dimensional point cloud obtainedfrom an optical scanning system and an associated outcome of a fittedsurface model.

FIG. 21 is a distance map wherein regions with high intensity indicatelarger computed distances.

FIGS. 22A and 22B are binary maps obtained via binary processing and anassociated filtered binary map indicating one or more possible polypsites areas and one or more non-possible polyp site areas.

FIG. 23 is a projection of a filtered distance map onto a threedimensional point cloud.

FIGS. 24A-24C illustrate an exemplary keypoint detection process inaccordance with the present disclosure.

FIG. 25 is a block diagram of an exemplary method of polyp detectionand/or measurement for a colonoscopy system having the optical scanningsystems of FIG. 18 or 19.

FIG. 26 is a three dimensional point cloud with data points located on apolyp border and lines connecting border data points to determineperimeter of a polyp.

FIGS. 27A-27C are data points describing exemplary methods fordetermining perimeter of a polyp in accordance with the presentdisclosure.

FIGS. 28A and 28B is an image of a mesh constructed from a threedimensional point cloud along a surface of a polyp and triangulation toaid in measuring surface area of the polyp, respectively.

FIG. 29 illustrates images associated with an exemplary method forregistration between two successive images.

FIGS. 30A and 30B illustrate images associated with an exemplary methodfor registration of blood vessel structures on a colon wall.

FIGS. 31A, 31B, 32A, and 32B illustrate an exemplary method forregistration in a colonoscopy system having the optical scanning systemsof FIG. 18 or 19.

FIGS. 33-34 illustrate an exemplary method for mapping and registrationbased on alternating images.

FIGS. 35A and 35B are graphs illustrating comparison of an estimatedpath and true path of an endoscope during a colonoscopy procedure.

FIGS. 36A-36I illustrate exemplary methods for using an exemplarycolonoscopy system in accordance with the present disclosure.

DETAILED DESCRIPTION

As described in further detail herein, in some embodiments,model-building capabilities of three-dimensional optical scanners andtwo-dimensional imaging methods are used, in addition to, novel polypdetection software, with all hardware miniaturized to fit within astandard endoscope's dimensions. In some embodiments, miniature laserarrays, pattern and solid illumination generation, NIR cameras, andadvanced processing algorithms are configured to meet size, mappingspeed, and accuracy needs of colonoscopy procedures.

In some embodiments, optical scanning systems may include one or morenear infrared (NIR) cameras and one or more NIR VCSEL sources integratedwithin an end of an endoscope (i.e., endoscopic hardware). Employing NIRsources and cameras takes advantage of the high NIR reflectivity oftissue to produce high resolution recordings without interfering withthe endoscope's existing visual systems. Each VCSEL source may occupy anarea less than 5 mm², for example. Each camera may occupy an area lessthan 12 mm², for example, such that components are configured to fitwithin the exemplary endoscope's 12.8 mm diameter. The 1- to 5-cmworking distance within the colon, combined with small component sizeand the capabilities of software, allow the system to operate on a 5.5mm baseline and still achieve sub-millimeter or millimeter measurementaccuracy in depth.

The optical sources may be configured to produce both patterned light(intensity variations over space) and solid illumination (no intensityvariation over space). Patterned light illumination of the colon wall,combined with stereoscopic vision provided by using one or more cameras,may produce data that allows the software to accurately locate each partof the colon in three-dimensional space, producing a three-dimensionalpoint cloud consisting of the collection of three-dimensional pointsidentified from the imaging data provided by the cameras.

The imaging and processing software may be configured to construct thethree-dimensional point clouds and three-dimensional models of colonsections with sub-millimeter or millimeter accuracy. Software algorithmsimplement polyp/adenoma detection functions that allow detection ofpolyps of any sizes including those equal to or below 6 mm, even whenpolyp/adenoma coloring closely matches the coloring of the colon wall.The solid illumination NIR source accentuates the contrast between bloodvessels and surrounding tissue in the colon wall providing a unique 2Dtopography, for example. As such, the software may be configured toperform pattern matching between images to extract registrationinformation (tilt and shift between successive images) to extractfeatures from the otherwise featureless colon wall and combiningthree-dimensional point clouds into a single, integrated threedimensional model of the entire colon, for example.

Combining three-dimensional and two-dimensional imaging capabilities mayallow the software to accurately locate polyps within the colon, detectcolon loops, detect when the operator does not fully scan a colonsection, precisely and/or accurately localize one or more tumors (e.g.,for follow-up surgical treatment), detect areas within the colon whereoperator missed areas of interest, and/or create a record for tracking apatient's colon health.

Before explaining at least one embodiment of the disclosure in detail,it is to be understood that the disclosure is not limited in itsapplication to the details of construction, experiments, exemplary data,and/or the arrangement of the components set forth in the followingdescription or illustrated in the drawings unless otherwise noted.

The disclosure is capable of other embodiments or of being practiced orcarried out in various ways. Also, it is to be understood that thephraseology and terminology employed herein is for purposes ofdescription, and should not be regarded as limiting.

The following detailed description refers to the accompanying drawings.The same reference numbers in different drawings may identify the sameor similar elements.

As used in the description herein, the terms “comprises,” “comprising,”“includes,” “including,” “has,” “having,” or any other variationsthereof, are intended to cover a non-exclusive inclusion. For example,unless otherwise noted, a process, method, article, or apparatus thatcomprises a list of elements is not necessarily limited to only thoseelements, but may also include other elements not expressly listed orinherent to such process, method, article, or apparatus.

Further, unless expressly stated to the contrary, “or” refers to aninclusive and not to an exclusive “or”. For example, a condition A or Bis satisfied by one of the following: A is true (or present) and B isfalse (or not present), A is false (or not present) and B is true (orpresent), and both A and B are true (or present).

In addition, use of the “a” or “an” are employed to describe elementsand components of the embodiments herein. This is done merely forconvenience and to give a general sense of the inventive concept. Thisdescription should be read to include one or more, and the singular alsoincludes the plural unless it is obvious that it is meant otherwise.Further, use of the term “plurality” is meant to convey “more than one”unless expressly stated to the contrary.

As used herein, any reference to “one embodiment,” “an embodiment,”“some embodiments,” “one example,” “for example,” or “an example” meansthat a particular element, feature, structure or characteristicdescribed in connection with the embodiment is included in at least oneembodiment. The appearance of the phrase “in some embodiments” or “oneexample” in various places in the specification is not necessarily allreferring to the same embodiment, for example.

Referring to the Figures, and in particular FIG. 1, shown therein anddesignated by reference numeral 10 is an exemplary colonoscopy system inaccordance with the present disclosure. Generally, the colonoscopysystem 10 is an intra-operative, dynamic three-dimensional scanningsystem configured to detect, measure and/or locate polyps and/oradenomas within a colon. In some embodiments, the colonoscopy system 10may include real time measurement data capable of providing guidanceand/or positioning of one or more tools for treatment and/or removal ofpolyps and/or adenomas of the colon. In some embodiments, thecolonoscopy system 10 may provide real-time scanning and processingsupports for augmented navigation guidance for optimal driving andpositioning of an endoscope. In some embodiments, the colonoscopy system10 may include real-time mapping and registration configured to providethree-dimensional model(s) supporting loop detection and/or monitoringof health of a patient. It should be noted that the colonoscopy system10 as described in detail herein may be a stand-alone system orintegrated with current standard colonoscopy systems known within theart.

The colonoscopy system 10 is configured to provide three-dimensionalpoint clouds with sub-millimeter or more (e.g., millimeter) accuracyduring a colonoscopy procedure and/or subsequent to the colonoscopyprocedure. The colonoscopy system 10 is configured to use uniquepatterns of blood vessels within the colon to extract information andposition from frame to frame and accurately stitch the three-dimensionalpoint clouds into a three-dimensional model of the colon. Additionally,by processing data within the three-dimensional point cloud and thethree-dimensional model, augmented navigation guidance may be providedallowing a user to position and/or direct the colonoscopy system 10within the colon. Such guidance may improve likelihood of full cecalintubation ensuring scanning of the entire colon and increase successrate of the colonoscopy procedure. Additionally such guidance mayprovide detection of colon loops, marking locations of cancerous tissuerequiring surgical removal and/or treatment, provide input data to oneor more control systems for semi-autonomous and autonomous colonoscopyprocedure, and/or detailed documentation of one or more steps of theprocedure providing information for continued prevention and/ortreatment of colorectal cancer and/or issues of the colon.

The colonoscopy system 10 is configured to detect and/or measure polypsand/or adenomas. Additionally, in some embodiments, the colonoscopysystem 10 is configured to alert an operator based on analysis ofgenerated three-dimensional point clouds.

Generally, the colonoscopy system 10 may be used as a stand-alone systemor a system integrated into currently used and future envisioned medicalsystems. In some embodiments, the colonoscopy system 10 may beintegrated within systems including, but not limited to, Olympus, BostonScientific, and Auris, for example. Additionally, the colonoscopy system10 may be adapted for use in other procedures including, but not limitedto, upper endoscopy, robotic or laparoscopic surgery autonomous andsemi-autonomous surgical procedures, surgical safety systems, and thelike.

Referring to FIG. 1, the colonoscopy system 10 includes an opticalscanning system 12, an optional attitude and heading reference system(AHRS) unit 14, and a control system 16. Generally, the optical scanningsystem 12 illuminates tissue in walls of a colon and captures one ormore images of the illuminated tissue. The AHRS unit 14 providesorientation and angular velocity (i.e., roll, pitch and yaw) data as afunction of time to assist mapping and registration functions. Thecontrol system 16 processes the one or more images captured by theoptical scanning system 12 and performs functions including, but notlimited to, (1) detection and/or measurement of polyps and/or adenomaswithin the colon, (2) construction of one or more three-dimensionalmodels and/or mappings of the colon for loop detection and otherfunctions, (3) augment existing displays with data, symbols, icons,and/or other indicators to direct an operating physician in treatingpolyps and/or adenomas and/or in driving an endoscope through the colon;(4) providing data for semi-autonomous or autonomous control and/oroperation of the endoscope during a colonoscopy procedure; (5) providingloop detection and/or treatment; and/or (6) providing detection oflocalized cancerous spots and/or un-visualized spots.

During a colonoscopy, generally an operating physician inserts aflexible endoscope (i.e., endoscopic hardware) into an anus and steersthe flexible endoscope to observe state of health of a colon andpossibly perform small surgical procedures to address problems such asthe presence of polyps and adenomas observed during the procedure. 3.The AHRS unit 14 provides orientation and angular velocity (yaw, pitch,and roll) data regarding the movement of the optical scanning system 12.The AHRS unit 14 includes, but is not limited to, (a) an inertialmeasurement unit (IMU) and an ASIC; (b) an IMU connected to an externalmicroprocessor by electronic cabling connecting the optical scanningsystem 12 to external power and processing systems, or the like.

Referring to FIGS. 2 and 3, in some embodiments, one or more opticalscanning systems 12 may mount on an outer wall an endoscopic instrument22 (e.g., near the head of the endoscope), as illustrated in FIGS. 2Aand 2B, and/or one or more optical scanning systems 12 may integratedirectly into hardware of an endoscopic instrument 22 a, as illustratedin FIG. 3. Endoscopic instrument 22 a may herein be referred to as‘endoscopic hardware’. In some embodiments, the optical scanning system12 may be mounted on a ring circumferentially fitted about an externalcircumference of a head of the endoscopic instrument. Further, cablingalong the internal and/or external length of the endoscope instrument 22may connect the optical scanning system 12 to one or more externalphysical and/or software components. It should be noted that the opticalscanning system 12 may maintain a wired and/or wireless connection toone or more external physical and/or software components including, butnot limited to, the AHRS unit 14, the control system 16, and/or anyother system regardless of whether the system is positioned on theendoscopic instrument 22 or 22 a, or positioned outside the body of apatient.

Each optical scanning system 12 may include one or more illuminator(s)18, one or more camera(s) 20, and a control system 26. Generally, theone or more illuminator(s) 18 includes one or more optically-basedpattern generator(s) 24 configured to impose spatial intensity and/orwavelength variation on light provided by one or more optical source(s)27. The camera 20 records an image of light projected onto surface oftissue, and the control system 26 synchronizes the optical source(s) 27,camera(s) 20 and optically based pattern generator(s) 24. It should benoted that the control system 26 may be integrated into control system16 based on design considerations.

The optical source(s) 27 may operate within infrared spectrum, UVspectrum and/or visible spectrum. For simplicity of description, thefollowing embodiments describe use of infrared spectrum, however, itshould be understood by one skilled in the art that an infrared, visibleor both visible and infrared optical source(s) 27 may be used within thecolonoscopy system 10. Further, the camera(s) 20 may be an infraredcamera, visible and/or multispectral cameras.

Generally, the optical scanning system 12 is configured to illuminateone or more areas of interest within the colon with one or more opticallight source(s) 27. Each optical source 27 may deliver significantoptical power to the tissue under investigation without causing damageto the target tissue or the patient due to heating or interactions withhigh optical intensities. In some embodiments, the optical source 27 mayinclude a laser diode, for example, operating in, but not limited to,the infrared region of the optical spectrum. Other diodes may be usedoperating in visible, or possibly both infrared and visible spectrums.For example, an operating region for the optical source 27 may be, butis not limited to, the near infrared (NIR) range between 700 nm and 1050nm. In some embodiments, wavelengths may include 780 nm, 808 nm, 850 nm,or 940 nm as these wavelengths are available in commercial sources,provide maximum optical reflection from biological tissue, and aresufficiently far from the visible light region of the optical spectrumto avoid interfering with a visible light camera that may be used byendoscopic systems (e.g., endoscopic instrument 22 illustrated in FIG.2) used in colonoscopy. The optical source 27 may provide sufficientpower for the reflection from the tissue to be captured with sufficientcontrast by the camera 20, but low enough power to avoid harm to thepatient or to saturate the sensors used in the camera 20.

In some embodiments, the optical source 27 may deliver optical power toa target tissue via an optical fiber, liquid light guide or similarwaveguide positioned about a side wall of an endoscopic instrument andterminating at the optical scanning system 12. In some embodiments, alarge-core optical fiber may be used. Such large-core optical fiber maybe configured to provide required power without damage to the opticalfiber and use a minimum of cladding to limit physical dimensions so asto fit within an existing endoscopic instrument (e.g., endoscopicinstrument 22 illustrated in FIG. 2). A large core optical fiber makesthe optical fiber highly multimode, ensuring that the power distributionover the optical fiber may be nearly uniform. In some embodiments, afused array of fibers or a shaped glass or plastic tube may be used.

In some embodiments, at least one end of the optical fiber may be formedas a flat surface or as a curved surface. A curved surface couldinclude, but is not limited to, having a spherical or parabolic shape,for example. A curved surface shape may enhance a range of angles overwhich the optical fiber illuminates the target area, thereby increasingthe size of the measurement area. A larger area may mean fewer imagesneeded and/or more overlapping sections between successive images (e.g.,improve stitching of images, improve accuracy/resolution).

In some embodiments, the optical source 27 may deliver optical power viaa laser diode source positioned at the optical scanning system 12,powered through electrical cable strung from an external power supplythrough or along the side of the endoscopic instrument to the opticalsource 27. Possible sources include, but are not limited to, anyinfrared laser diode, including an edge-emitting laser (EEL) or avertical cavity surface emitting laser (VCSEL). The VCSEL-based sourcemay include, but is not limited to, a single VCSEL or a patterned arrayof VCSEL sources, the latter of which contributes directly to thegeneration of spatial intensity patterns on the tissue underinvestigation. In some embodiments, the optical source 27 and/or opticalscanning system 12 may include mechanisms for mitigating heat.

Referring to FIGS. 4A and 4B, in some embodiments, the illuminator 18may include the optical source 27 and the optically-based patterngenerator having an optical imaging system 28 and/or a diffractiveoptical element 30. It should be noted that in some embodiments, theilluminator 18 may solely include the optical source 27.

The optical imaging system 28 collects light from the optical source 27and controls divergence and convergence of optical beams as each opticalbeam passes through the optically-based pattern generator 24. Theoptical imaging system 28 may include, but is not limited to, a singleconvex lens or a combination of two or more convex and/or concavelenses, projection optics, micro-optical elements, single opticalelement(s), meniscus lens, or combinations of two or more opticalelements, convex lens, meniscus lens and/or concave lenses. In someembodiments, the focal length of the optical imaging system 28 may beselected to produce a primary imaging point at the average distancebetween the illuminator 18 and/or camera 20 and a wall of the colon, forexample, typically around 3 cm, and to produce a long depth of focus, insome embodiments extending between 1 cm and 5 cm, to allow projection ofsharp, high-contrast images for a range of distances between the opticalscanning system 12 and/or camera 20 and the colon wall. In someembodiments, focal lengths minimize the distance between the opticalsource 27 and the optical imaging system 28 to minimize the size (indepth) of the illuminator 18 to facilitate integration with or mountingon the endoscope (e.g., endoscope instrument 22 illustrated in FIG. 2).

The optically-based pattern generator 24 may be configured to providestructured light and/or unstructured (i.e., unpatterned, flood, flat-topillumination or solid) illumination, and/or combinations thereof. Forclarity in description, a structured light source is herein defined as asource configured to produce an optical light beam containing purposefulspatial variations in optical intensity. An unstructured light source,flood, flat-top illumination pattern or solid illumination light sourceis defined as a source configured to produce nearly constantillumination intensity over an area or range of angles. Generally, theoptically-based pattern generator 24 may impose spatial intensityvariation on the optical beam produced by the combination of the opticalsource 27 and the optical imaging system 28.

Referring to FIGS. 4A and 4B, in some embodiments, the optically-basedpattern generator 24 may include the diffractive optical element (DOE)30 positioned after the optical imaging system 28 at a distance selectedto produce a specific pattern with specific spatial variation at theprimary imaging point, including composite images consisting of bothpatterned and unpatterned sections, as shown in FIGS. 4A and 4B for boththe edge-emitting laser (EEL) and vertical cavity surface emitting laser(VCSEL) optical sources 27. In some embodiments, the optically-basedpattern generator 24 may include the diffractive optical element (DOE)positioned before the optical imaging system 28. The DOE 30 may include,but is not limited to, (a) surface height variations, such as etchedgratings, and/or (b) variations in refractive index within the basematerial, such as holographic elements. For the VCSEL-based embodimentin FIG. 4B, the optical source 27 includes an array of VCSELs, with thearrangement of VCSELs chosen in combination with the DOE 30 to produce apattern consisting of the convolution between the patterned-VCSEL arrayand the spatial variations in the DOE 30.

Referring to FIGS. 5A-5C, in some embodiments, the illuminator 18includes optical source 27 that is an array of VCSELs, optical imagingsystem 28 and/or diffractive optical element 30 and illuminates thetissue. In some embodiments, the optical source 27 as an array of VCSELsis driven by a constant drive current, producing almost the same opticalpower over the full emitted optical beam, and the pattern of lightilluminating the tissue is created by the special variations in thediffractive optical element 30. Solid (unpatterned) light results from aspatial variation in the diffractive optical element 30 thatredistributes the optical power from the optical source 27 as the arrayof VCSELs such that the optical power remains constant or nearlyconstant over the illuminated area as shown in FIG. 5A. Composite imagescontaining both patterned and unpatterned components result from aspatial variation in the diffractive optical element 30 which producesconstant or nearly constant power (unpatterned) over some fraction ofthe illuminated area and produces specific variations (patterned) inoptical power over remaining fraction of the illuminated area as shownin FIG. 5C. An alternative method of producing solid, patterned, orcomposite illumination include controlling the drive current to eachVCSEL individually, thereby controlling the optical power output at eachpoint in the VCSEL array. Solid (unpatterned) light results fromoperating all of the VCSELs at the same optical power as shown in FIG.5A. Composite images containing both patterned and unpatternedcomponents result from operating section of the VCSEL array at the sameoptical power and operating other sections of the VCSEL array withoptical powers that vary spatially over the array as shown in FIG. 5C.

Optical patterns produced by the optically based optically-based patterngenerator 24 may include, but are not limited to, a set of highresolution optical patterns, binary patterns, gray patterns, phase shiftpatterns, hybrid gray and phase shift patterns, rainbow patterns,continuously varying color patterns, color coded stripes, wavelengthcoded stripes, wavelength pattern, segmented stripes, gray scale codedstripes, De Bruijin Sequence, Pseudo Random Binary dots, mini-patternsas codewords, color coded grids, two dimensional coded, two dimensionalcoded dot array, and/or any combination thereof. Exemplary patterns andassociated measurement techniques may be found in the article by JasonGeng, Structured-light 3D Surface Imaging: a tutorial, Advances inOptics and Photonics 3, 128-160 (2011), which is hereby incorporated byreference in its entirety.

Referring to FIG. 6A, illustrated therein is composite frame 40 providedby the illuminator 18 having sides 42, 44, 46 and 48. The compositeframe 40 consists of at least one section with patterned illumination 50and at least one section of unpatterned (solid) illumination 52. In oneexample as illustrated in FIG. 6A, a the patterned illumination 50 ispositioned within in a center of the composite frame 40 and unpatternedillumination 52 is positioned about and extending along the sides 42,44, 46 and 48 as illustrated in FIG. 6A. With regards to the patternillustrated in FIG. 6A, the patterned illumination may be used forconstructing three-dimensional point clouds and the solid illuminationmay be used for registration and mapping as described in further detailherein. The patterned infrared illumination constitutes 50% of theilluminated area and the solid (unpatterned) illumination constitutes50% of the illuminated area in FIG. 6A; however, other combinations ofthe illuminated areas are possible.

Referring to FIG. 6B, illustrated therein is composite frame 40 a havingsides 42 a, 44 a, 46 a and 48 a. The composite frame 40 a consists of atleast one section with patterned illumination 50 a and at least onesection of unpatterned (solid) illumination 52 a. In one example, asillustrated in FIG. 6B, a corner area pattern wherein the unpatternedillumination 52 a is limited to corners of the frame between sides 42 aand 44 a, and sides 46 a and 48 a with the patterned illumination 50 aconstituting the remaining area of the composite frame 40 a. In FIG. 6B,unpatterned illumination 52 a comprises just about 8% of the total framearea, contain solid (unpatterned) illumination. In FIG. 6C, where fourcorners between sides 42 b, 44 b, 46 b and 48 b, comprising less than20% of the total area of the composite frame 40 b, contain unpatternedillumination 52 b.

The optical scanning system 12 records images of the light beamsprojected onto the tissue of the colon wall using one or more camera(s)20, e.g., high-resolutions infrared camera(s). Each recorded imagerepresents one frame captured at a specific time. During the scanningprocess, the illuminator 18 may illuminate the structure (e.g., areawithin the colon) with one or more different images or frames (i.e.,multi shots such as binary code, gray code, phase shift code, hybrid ofgray code and phase shift code, other hybrids, and/or the like), orsingle image or frame (i.e., single shot such as color coded stripes,wavelength coded stripes, wavelength pattern, segmented stripes, grayscale coded stripes, De Bruijin sequence, pseudo random binary dots,mini-patterns as codewords, color coded grid, two dimensional colorcoded dot array, hybrids, and/or the like). The illuminator may projecta structured-light image, an unpatterned (solid) illumination, or acomposite image containing both patterned and unpatterned sectionsduring the capture of each frame.

The camera 20 may possess responsivity to infrared, visible or bothvisible and infrared light. Additionally, the camera 20 may beconfigured to produce high-resolution images of the projected patternand have both length and width dimensions that minimize the dimensionsof the optical scanning system 12. In some embodiments, the camera 20may include a mono sensor with a pixel size between 1 μm and 1.4 μm andan array of 1900×1900 pixels. The pixel size determines the focal lengthof the optical scanning system 12 in pixels, according to EQ. 1.

$\begin{matrix}{f_{p} = \frac{f_{mm}}{p_{mm}}} & \left( {{EQ}.\mspace{14mu} 1} \right)\end{matrix}$

wherein f_(p) is the focal length in pixels, f_(mm) is the focal lengthof the optical scanning system 12 in millimeters, and p_(mm) is thepixel size in millimeters. In some embodiments, sensor dimensions do notexceed 4.5 mm×3.5 mm to allow sufficient space for the components of theilluminator 18 and to allow for sufficient separation between the camera20 and the illuminator 18 to achieve sub-millimeter or millimeteraccuracy from the triangulation algorithms implemented in the softwareprocessing systems as described in further detail herein.

In some embodiments, a camera optics system 29 may be positionedadjacent to the camera 20 to facilitate light collection and/or providethe proper optical geometry for achieving high-accuracy opticalscanning. The camera optics system 29 may include, but is not limited toa single optical element, convex lens, meniscus lens, concave lens, orcombination of two or more optical elements, convex lens, meniscus lensand/or concave lens. In some embodiments, the camera optics system 29used with the camera 20 may be substantially similar to or identical tothe optical imaging system 28 used with the optical source 27 (e.g.,when using a single camera with a single structured light source).

Referring to FIGS. 1, 5A-5C and 6A-6C, in some embodiments, the controlsystem 26 and/or control system 16 synchronizes the optical source 27,optically-based pattern generator 24, and camera 20 to produce andcapture illumination patterns 50 and 52 produced by the illuminator 18,varies illumination patterns 50 and 52 between frames 40, and maycapture orientation and angular momentum data from the AHRS unit 14. Thecontrol system 26 includes, but is not limited to, a source controller54 and a synchronization subsystem 56.

The source controller 54 ensures that the correct drive current(s) areprovided to the optical source 27 during each frame 40 captured by theoptical scanning system 12. For example, for a single laser diode or astatic VCSEL array source as illustrated in FIG. 4, the sourcecontroller 54 maintains a desired level of current to the laser diode orto each VCSEL in the array. For the embodiment in which the sourceoutput alternates between patterned illumination and unpatterned (solid)illumination, for example for the source depicted in FIGS. 5A-5C, thesource controller 54 may select the correct set of drive currents forthe corresponding set of VCSELs in the array to produce the desiredillumination from the optical source 27. The source controller 54 mayinclude, but is not limited to, a microprocessor, electronic drivecircuitry for precise control of current delivered to the source, anddigital logic or switches for switching between two or more illuminationpatterns.

The synchronization subsystem 56 ensures that camera 20 captures animage only after the desired illumination pattern 50 and/or 52 isproduced by the illuminator 18. The synchronization subsystem 56 alsoprovides timing signals to the AHRS unit 14 to ensure that orientationand angular momentum data is captured at the exact same time as thecapture of the image by the camera 20. The synchronization subsystem 56may include, but is not limited to, a microprocessor and a timing signalgeneration system.

The geometrical arrangement of the illuminator 18 and/or camera 20 maydetermine design of size and resolution of the optical scanning system12. Further, resolution impacting the measurement accuracy achievedafter the processing software produces the 3D point cloud may beaffected by geometrical arrangement of the illuminator 18 and/or camera20. Described herein are exemplary parallel configurations and exemplaryconverged configurations; however, descriptions of each are not to beconsidered limiting as other configurations may be contemplated by oneskilled in the art and within the bounds of teachings within thedescription (e.g., configurations that alter size, resolution of theoptical scanning system and configurations that impact measurementaccuracy).

FIGS. 7A and 7B illustrate an exemplary optical scanning system 12having a parallel configuration 60 of the illuminator 18 and the camera20 wherein the illuminator 18 and the camera 20 are along a plane P₁ andhave an orientation perpendicular to the surface of the plane P₁. Forexample, the illuminator 18 is oriented in the direction of plane P₃with plane P₃ positioned perpendicular to plane P₁. Similarly, thecamera 20 is oriented in the direction of plane P₂, with the plane P₂positioned perpendicular to plane P₁. Referring to FIG. 7B, for theparallel configuration 60, an overlap area OV₁ of tissue is illuminatedby the illuminator 18 within the field of view FOV₁ of the camera 20. Insome embodiments, the illuminator 18 illuminates an area of tissuewithin as much of the FOV₁ of the camera 20 as possible. Increasing theoverlap area OV₁ reduces the minimum depth from which the opticalscanning system 12 can obtain data about the illuminated tissue of thewall of the colon.

For the camera 20, the FOV₁ is determined by sensor dimensions and focallength of the camera optics system 29 placed in front of the camera 20.The FOV₁ depends on the angle of view (α), defined as the angle overwhich the sensor can collect light from the scene. The angle of view, inradians, is given by:

$\begin{matrix}{\alpha = {2*{\tan \left( \frac{W\left( {s - f_{mm}} \right)}{2sf_{mm}} \right)}}} & \left( {{EQ}.\mspace{14mu} 2} \right)\end{matrix}$

wherein W is the width of the sensor used in the camera 20, s is thedistance between the subject (e.g., wall tissue of the colon) and thesensor, and f_(mm) is the focal length of the camera optics system 29.The FOV₁ depends on the angle of view (AOV) according to:

$\begin{matrix}{{FOV}_{1} = {2*{\tan \left( \frac{\alpha}{2} \right)}*d}} & \left( {{EQ}.\mspace{14mu} 3} \right)\end{matrix}$

wherein d is the distance (or depth) to the tissue from the camera 20.

The parallel configuration 60 shown in FIG. 7A may additionally beinfluenced by measurements of baseline L and end-to-end length EE.Baseline L is defined as distance between center C₁ of the camera 20 andcenter C₂ of the illuminator 18. The end-to-end length EE is defined asdistance between an outer edge 64 of the camera 20 and an outer edge 66of the illuminator 18. In some embodiments, the end-to-end length EE maybe limited to less than approximately 12 mm. Accuracy in depth d_(a)depends on the operating distance h between the optical scanning system12 and the tissue under investigation, the focal length f of the opticalimaging system 28, and the distance of the baseline L according to theformula:

$\begin{matrix}{d_{a} = {\frac{h^{2}}{Lf}c_{e}}} & \left( {{EQ}.\mspace{14mu} 4} \right)\end{matrix}$

wherein c_(e) is a calibration and matching error correction factor (inpixels) in the processing software. As an example, if the camera 20possesses 1920×1080 pixels with pixel dimension of 1.4 μm, combined withthe optical scanning system 12 having f=1.83 mm, a baseline L=4 mm, andpackage dimensions of 4.1 mm×3.9 mm produces a depth accuracy of 1.5 mmat a nominal h=2.5 cm and allows an end-to-end length EE of 8 mm. Theaccuracy in the lateral direction (e_(l)) is determined by:

$\begin{matrix}{e_{l} = {\frac{h}{f_{p}}e_{a}}} & \left( {{EQ}.\mspace{14mu} 5} \right)\end{matrix}$

wherein f_(p) is the focal length in pixels, and e_(a) is the algorithmerror in pixels, and is typically fixed at 0.5 times the pixel meanerror.

FIGS. 8A and 8B illustrate an exemplary converged configuration 62 ofthe illuminator 18 and the camera 20 wherein the illuminator 18 and/orthe camera 20 are each tilted at an angle about the planes P₂ and P₃respectively. In some embodiments, only one of the camera 20 or theilluminator 18 may be tilted about the plane P₂ and P₃ respectively.Converged configurations 62 may include, but are not limited to, (a)physically tilting the camera 20 and the illuminator 18 toward eachother at some angle (hardware tilting as shown in FIG. 8A), or (b)designing the optical imaging system 28 to tilt the FOV and theprojected beam toward each other, with the camera 20 and the illuminator18 remaining physically parallel to each other (optical tilting as shownin FIG. 8B).

For the converged configuration 62, the tilting angle and the AOVdetermine an overlap area OV₂. If the tilting angle is less than theAOV, the overlap area OV₂ possesses a minimum depth from which thecamera 20 can collect image data, as shown in FIG. 9A. FIGS. 9A and 9Billustrate instances wherein the tilting angle is respectively greaterthan and less than the AOV. As shown, the overlap area OV₂ possessesboth a minimum and a maximum depth from which the camera 20 can collectimage data, as shown in FIG. 9B. For a given AOV and baseline distance,the minimum depth achieved by the converged configuration 62 may be lessthan the minimum depth achieved by the parallel configuration 60. TheAOV and FOV of the camera 20, the depth accuracy, and the lateralaccuracy may be determined using EQ. 2-5 described herein.

FIGS. 10-19 illustrate exemplary embodiments of the colonoscopy systems10 in accordance with the present disclosure.

FIG. 10 illustrates an exemplary colonoscopy system 10 with one camera20 and at least one structured illuminator 18. The camera 20 may be ahigh-resolution camera sensitive to infrared light and configured torecord at least one image of light projected onto a surface of tissueunder investigation. Additionally, the colonoscopy system 10 may includeat least one optical imaging system 28 that imposes a spatial intensityand/or wavelength variation on light from at least one illuminator 18,and the electronic control system 26 configured to synchronize one ormore optical sources 27, the optically-based pattern generator 24, andthe camera 20. As described in further detail here, properties,arrangement and/or configuration of the illuminator 18, optically-basedpattern generator 24, camera 20 and/or the like within the colonoscopysystem 10 a and further within mechanical housing encasing thecomponents (e.g., ring about the endoscopic instrument 22, internal tothe endoscopic instrument 22 as shown in FIGS. 2 and 3) may determineperformance of the colonoscopy system 10 in terms of lateral and depthresolution, depth of tissue for which the target resolution is achieved,and the field of view over which the system can make measurements.

The control system 26 may be configured to synchronize the opticalsource 27, optically based pattern generator 24, and camera 20 toproduce and capture illumination patterns produced by the illuminator18, vary illumination patterns between frames 40 (shown in FIGS. 5-6),and capture orientation and angular momentum data from the AHRS unit 14.

The control system 16 may associate components of the pattern recordedin the image obtained by the camera 20 with the corresponding point(s)in the original projected pattern, and measure the translation androtation of the optical scanning system 12 between successive frames.Using the data, the control system 16 may construct one or morethree-dimensional point clouds and perform registration calculation tostitch the 3D point clouds into a mapping of the tissue illuminated bythe illuminator 18. Outputs of the control system 16 may include, butare not limited to, alert, detection and/or identification of polyps andadenomas, measurements of height and circumference of the polyps andadenomas, measurement data and other informational icons for augmenteddisplays, three-dimensional models of the scanned areas or lengths ofthe colon, and control signals for semi-autonomous and autonomousoperation of the endoscopic system 22.

Referring to FIGS. 5, 6 and 10, the exemplary colonoscopy system 10, theilluminator 18 illuminates tissue in the walls of the colon withstructured light. In this context, structured light consists of lightwith regular and controlled spatial variation in intensity, typicallyreferred to as spatial patterns. Such optical patterns may include, butare not limited to, a set of high resolution optical patterns, binarypatterns, gray patterns, phase shift patterns, hybrid gray and phaseshift patterns, rainbow patterns, continuously varying color patterns,color coded stripes, segmented stripes, gray scale coded stripes, DeBruijin Sequence, Pseudo Random Binary dots, mini-patterns as codewords,color coded grids, two dimensional coded dot array, and/or anycombination thereof. Exemplary patterns and associated measurementtechniques may be found in the article by Jason Geng, Structured-light3D Surface Imaging: a tutorial, Advances in Optics and Photonics 3,128-160 (2011), which is hereby incorporated by reference in itsentirety. The structured light of the illuminator 18 may further consistof combinations of an area containing one of the potential patternedillumination 50 with one or more areas containing unpatternedillumination 52 (i.e., solid illumination), as shown in FIGS. 5-6.

During the scanning process, the illuminator 18 may illuminate with oneor more different images or frames 40 (i.e., multi shots such as binarycode, gray code, phase shift code, hybrid of gray code and phase shiftcode, other hybrids, and/or the like), or single image or frame 40(i.e., single shot such as color coded stripes, segmented stripes, grayscale coded stripes, De Bruijin sequence, pseudo random binary dots,mini-patterns as codewords, color coded grid, two dimensional colorcoded dot array, hybrids, and/or the like). For some embodiments, theilluminator 18 may alternate between illumination of the colon wall, forexample, with patterned illumination 50 and unpatterned illumination 52,with the alternation occurring between successive frames 40 and/or timeintervals. The camera 20 may be a single, high-resolution cameraconfigured to capture one or more images of the colon tissue, forexample, illuminated by the illuminator 18, with the image showing theprojection of the patterned illumination 50 and/or unpatternedillumination 52 onto the three-dimensional space.

For frames 40 that include patterned illumination 50 or patternedcomponents of composite frames 40, a matching operation may beperformed, wherein each part of the projected pattern is matched to acomponent of the original pattern stored in memory. As such, adetermination may be made on which portion of the original patternilluminated each section of tissue within the colon. Matching data,along data related to geometrical arrangement of the camera 20 and theilluminator, may be input into sophisticated triangulation algorithms.The triangulation algorithms use the information to calculate a locationin 3D space for each segment of the colon tissue. Repeating the processfor two different patterns projected on the same section of tissueincreases the accuracy of the triangulation process and allows thecolonoscopy system 10 to produce highly accurate three dimensional pointcloud representation of the illuminated tissue.

For the frames 40 having unpatterned illumination 52 or unpatternedcomponents of composite frames 40, a matching operation may be performedbetween the patterns of blood vessels contained in each frame 40,wherein specific features or patterns of blood vessels are matchedbetween successive frames by global or semi-global registrationtechniques. Blood vessel matching information may be used in addition tothe 3D point clouds from the patterned frames, and the orientation andangular velocity data collected from the AHRS unit 14 at every frame, toperform registration between the 3D point clouds and subsequentlyconstruct a 3D model of the colon wall.

FIGS. 11-13 illustrate exemplary embodiments of the colonoscopy system10 b that include the illuminator 18 and two cameras 20 a and 20 blocated at two different positions with respect to the illuminator 18.The geometrical arrangement of the illuminator 18 and the cameras 20 aand 20 b determines size and resolution of the optical scanning system12 b. Resolution may impact measurement accuracy achieved after theprocessing software produces the three dimensional point cloud.Generally, the cameras 20 a and 20 b are positioned at either end of theoptical scanning system 12 b, and the illuminator 18 is placed in closeproximity to the cameras 20 a and 20 b, with the illuminator 18positioned along the same plane to minimize size of the optical scanningsystem 12 b as illustrated in FIG. 11. This configuration between theilluminator 18 and two cameras 20 a and 20 b is herein referred to as“stereo configuration”. FIG. 12 illustrates another implementation of“stereo configuration” wherein the illuminator 18 is positioned betweenthe cameras 20 a and 20 b. In FIG. 12, the illuminator 18 is positionedat a side of the cameras 20 a and 20 b.

Generally, the control system 16 performs pattern matching and stereotriangulation between the two cameras 20 a and 20 b. Additionally,translation and rotation of the optical scanning system 12 b is measuredbetween successive frames to construct three-dimensional point cloudsfrom the patterned illumination data. Additionally, registrationcalculation is performed to the stitch the three-dimensional pointclouds into a mapping of the tissue illuminated by the illuminator 18.Outputs may include, but are not limited to, alert, detection and/oridentification of polyps and adenomas, measurements of height andcircumference of the polyps and adenomas, measurement data and otherinformational icons for augmented displays, three dimensional models ofthe scanned areas or lengths of the colon, and control signals forsemi-autonomous and autonomous operation of the endoscopic system.

The geometrical arrangement of the optical scanning system 12 b mayinclude, but is not limited to a parallel configuration and a convergedconfiguration as described in further detail in FIGS. 7-9 herein. Thegeometric arrangement of the two cameras 20 a and 20 b may be such thatthe illuminator 18 produces a structured light beam that illuminates asmuch of the combined FOV₃ of the cameras 20 a and 20 b as possible tomaximize the area of illuminated tissue recorded.

Referring to FIGS. 12 and 13, geometric design of the optical scanningsystem 12 may consider total stereo FOV₃, defined as the mutual FOV thecameras 20 a and 20 b or the overlap between the individual camera FOVs,as an additional parameter. Computing the mutual FOV₃ at a given depthrequires four data points, as shown in FIG. 14. The four points are (1)LA, the left most point of the FOV_(a) of the camera 20 a at a chosendepth; (2) LB, the right most point of FOV_(a) of the camera 20 a at achosen depth; (3) RA, the left most point of the FOV_(b) of the camera20 b at a chosen depth; and (4) RB, the right most point of the FOV_(b)of the camera 20 b at a chosen depth. Such points may be determined viaEQS. 6-9:

$\begin{matrix}{{LA} = {{\tan \left( \frac{{- \alpha} + \theta}{2} \right)} \cdot d}} & \left( {{EQ}.\mspace{14mu} 6} \right) \\{{LB} = {{\tan \left( \frac{\alpha + \theta}{2} \right)} \cdot d}} & \left( {{EQ}.\mspace{14mu} 7} \right) \\{{RA} = {{{\tan \left( \frac{{- \alpha} - \theta}{2} \right)} \cdot d} + b}} & \left( {{EQ}.\mspace{14mu} 8} \right) \\{{RB} = {{{\tan \left( \frac{\alpha - \theta}{2} \right)} \cdot d} + b}} & \left( {{EQ}.\mspace{14mu} 9} \right)\end{matrix}$

wherein α is the FOV, θ is the stereo tilt angle, d is the depth inmillimeters and b is the baseline between the cameras 20 a and 20 b inmillimeters. The mutual or stereo FOV₃ can be calculated for the twocases shown in FIGS. 15A and 15B. For FIG. 15A, the stereo FOV₃ equalsLB−RA. For FIG. 15B, the stereo FOV₃ equals RB−LA.

Referring to FIGS. 16A-16C, in one example, an OV02281-GA4A sensor isused in the camera 20 a, which has a pixel size of 1.12 μm, an arraysize of 2.214×2.214 mm, and a package size of 4050×3400.2 μm. Theoptical scanning system 12 b has a focal length of 2 mm with the cameras20 a and 20 b. The optical scanning system 12 b also includes a baselineof 7.95 mm and stereo tilt angle of 14° between the cameras 20 a and 20b. At a depth of 25 mm, the resultant stereo FOV₃ width is 26 mm, over99.5% of the FOV is used in capturing the image projected by theilluminator 18, the depth error is 22 μm, the lateral error is 14 μm,and the cameras 20 a and 20 b can capture images for all depths beyond6.8 mm (i.e., the depth range is 6.8 mm to infinity). FIGS. 16A-16B aregraphs 70, 72 and 74 that show the dependence of the FOV₃, depth error,and lateral error on the depth, respectively, between the cameras 20 aand 20 b and the tissue wall of the colon.

Referring to FIGS. 5-6 and 11-12, for the patterned frames or patternedcomponents of composite frames 40, a matching operation may be performedwherein a determination is made as to the position of the same componentof the projected pattern within both of the captured images, using theoriginal pattern stored in memory. Location of each pattern component inthe two images may be used and information on the geometry between thetwo cameras 20 a and 20 b may be input into the triangulationalgorithms. The triangulation algorithms use the information tocalculate a location in three-dimensional space for each segment of thecolon tissue, and subsequently produce a highly accuratethree-dimensional point cloud representation of the illuminated colontissue.

For the unpatterned (solid) frames or unpatterned components ofcomposite frames 40, a matching operation may be performed between thepatterns of blood vessels in each frame 40, using data from the imagescaptured by both cameras 20 a and 20 b. Specific features or patterns ofblood vessels are matched between successive frames 40 by global orsemi-global registration techniques. Blood vessel matching information,the three-dimensional point clouds from the patterned frames 40, and theorientation and angular velocity data collected from the AHRS unit 14 atevery frame, may be used to perform registration between thethree-dimensional point clouds and subsequently construct athree-dimensional model of the colon wall.

FIG. 17 illustrates an exemplary embodiment of the colonoscopy system 10c that includes two illuminators 18 a and 18 b, and at least one camera20 to produce image data for processing by the control system 16. Theilluminators 18 a and 18 b may be comprised of a structured light sourceand a solid light source (i.e., flood source) as defined herein.

Generally, the colonoscopy system 10 c includes, but is not limited to,subsystem one which now consists of the two illuminators 18 a and 18 bhaving one or more optically-based pattern generators 24 applied to atleast one optical source 27 a and/or 27 b, at least one camera 20 (e.g.,high resolution camera), and at least one control system 26. Theproperties of the components within the illuminators 18 a and 18 b andthe camera 20, the arrangement of the with respect to each other and/orthe configuration of the optical scanning system 12 c within housingencasing the components on or within the endoscopic system 22 maydetermine the performance of the colonoscopy system 10 c in terms oflateral and depth resolution, the depth of tissue for which the targetresolution is achieved, and the field of view over which the colonoscopysystem 10 c can make measurements.

At least one of the components of the illuminator 18 a or 18 b (e.g.,infrared light source), may be separate from the other illuminator 18 aor 18 b (e.g., structured light source), and illuminate the tissue underinvestigation with a solid or uniform intensity. In some embodiments,the illuminator 18 a or 18 b having the structured light source and theilluminator 18 a and 18 b providing solid illumination may illuminatethe tissue in alternating frames 40, with one of the illuminators 18 aor 18 b providing illumination and the other illuminator 18 a or 18 bturned off in one frame 40, and the alternate illuminator 18 a or 18 bturned on. In some embodiments, both of the illuminators 18 a and 18 bmay illuminate the tissue in one frame 40, with the illuminator 18 a or18 b providing solid illumination illuminating the tissue in thesubsequent frame 40.

The geometrical arrangement of the illuminators 18 a and 18 b, and thecamera(s) 20 may determine the size and resolution of the opticalscanning system 12 c, with the resolution impacting the measurementaccuracy achieved after the processing software produces thethree-dimensional point cloud. Referring to FIG. 17, the camera(s) 20may be positioned at either end 80 a and 80 b of the optical scanningsystem 12 c. At least one illuminator 18 a or 18 b (i.e., providingstructured light) may be positioned in close proximity to the camera(s)20. The illuminator 18 a or 18 b providing solid illumination may bepositioned between the cameras 20 and next to the illuminator 18 a or 18b providing structured light, outside the cameras 20, or outside thebaseline between the camera(s) 20 and the illuminator 18 a or 18 bproviding structured light, for example, in an orthogonal direction tothe that of the baseline. Other positioning is herein contemplated. FIG.17 illustrates an exemplary embodiment with the illuminator 18 aproviding structured light and the illuminator 18 b providing solid(i.e., flood) illumination positioned between two cameras 20 along thesame plane to minimize the size of the optical scanning system 12 c.

The geometrical arrangement of the optical scanning system 12 c mayinclude, but is not limited to a parallel configuration and a convergedconfiguration as described previously herein. In both the parallelconfiguration and converged configuration, the geometric arrangement ofthe two cameras 20 may be the primary concern. The illuminator 18 a or18 b providing structured light is configured to produce a structuredlight beam that illuminates as much of the combined FOV of the cameras20 as possible to maximize the area of illuminated tissue recorded. Theilluminator 18 a or 18 b providing solid (flood) light is alsoconfigured to produce a constant or nearly constant illuminationintensity that illuminates as much of the combined FOV of the cameras 20as possible to maximize the ability of the system to accurately performregistration operations. In some embodiments, the illuminator 18 a or 18b providing solid (flood) light may illuminate an area larger than thecombined FOV to provide sufficient data to produce accurateregistration.

Referring to FIGS. 1, 4, and 17, the illuminator 18 a or 18 b includesthe optical imaging system 28 and optical source 27. The optical imagingsystem 28 collects the light from the optical source 27 and controls thedivergence of the optical beam for each illuminator 18 a and 18 b. Theoptical imaging system 28 used for the illuminator 18 a or 18 bproviding structured light may be different in components, structure,and performance parameters than the illuminator 18 a or 18 b providingsolid illumination. The optical imaging system 28 may include, but isnot limited to, a single convex lens or a combination of two or moreconvex and/or concave lenses. For the structured light source, thedesign of the optical imaging system 28 may be selected such that thefocal length of the optical imaging system 28 may produce a primaryimaging point at the average distance between the optical scanningsystem 12 c and the colon wall, typically around 3 cm, and may produce along depth of focus, (e.g., extending between 1 cm and 5 cm), to allowprojection of sharp, high-contrast images for a range of distancesbetween the optical scanning system 12 c and the colon wall. Exemplaryfocal lengths for use may be configured to minimize the distance betweenthe illuminator 18 a or 18 b and the optical imaging system 28 tominimize the size (in depth) of the optical scanning system 12 c tofacilitate integration with or mounting on the endoscope 22. For theilluminator 18 a or 18 b providing solid illumination, the design may beconfigured to project a constant or nearly-constant illumination overthe tissue that falls within the field of view (FOV) of the camera(s)20, in order to provide sufficient detail to maximize the performance ofthe mapping, registration and modeling software system, for example.

The illuminator 18 a or 18 b providing structured light projectspatterns onto the colon wall, wherein the patterns may consist of, butare not limited to, arrays of dots, lines, and other geometric figures,and may or may not also contain color variations. The illuminator 18 aor 18 b providing solid (unpatterned) light illuminates the colon wallwith a uniform illumination. The two illuminators 18 a and 18 billuminate the colon wall in alternating frames or time intervals. Oneor two cameras 20 capture an image of the colon tissue illuminated bythe illuminator 18 a or 18 b providing structured light or theilluminator 18 a or 18 b providing solid (unpatterned) light, with theimage showing the projection of the optical pattern or solidillumination onto the three-dimensional space.

For the patterned frames 40, the software first performs a matchingoperation, wherein each part of the projected pattern is matched to acomponent of the original pattern stored in memory. In this way, thesoftware determines which part of the original pattern illuminated eachsection of tissue within the colon.

Specialized analysis software employs triangulation algorithms toassociate components of the structured-light pattern recorded in theimage of the camera 20 with the corresponding point in the originalprojected pattern and constructs three-dimensional point clouds of thetissue illuminated by the illuminator 18 a or 18 b providing structuredlight. To that end, the software uses the matching information, alongwith information about the geometrical arrangement of the camera 20 andilluminator 18 a and/or 18 b, as input to sophisticated triangulationalgorithms. The triangulation algorithms use the information tocalculate a location in three dimensional space for each segment of thecolon tissue. Repeating the process for two different patterns projectedon the same section of tissue increases the accuracy of thetriangulation process and allows the system to produce highly accuratethree-dimensional point cloud representation of the illuminated tissue.

The software uses the blood vessel matching information, the threedimensional point clouds from the patterned frames 40, and theorientation and angular velocity data collected from the AHRS unit 14 atevery frame 40, to perform registration between the three dimensionalpoint clouds and subsequently construct a three dimensional model of thecolon wall. To that end, for the unpatterned (solid) frames 40, thesoftware performs a matching operation between the patterns of bloodvessels contained in each frame 40, where specific features or patternsof blood vessels are matched between successive frames by global orsemi-global registration techniques. The software further employsalgorithms for matching the patterns of blood vessels recorded by thecamera 20 in successive frames in response to the illuminator 18 a or 18b providing solid illumination, performs registration and stitchestogether the three-dimensional point clouds to produce three dimensionalmappings of the tissue illuminated by the illuminator 18 a or 18 b.Additional algorithms utilize the three-dimensional point clouds andthree dimensional mapping to produce outputs that include, but are notlimited to, detection and identification of polyps and adenomas,measurements of height and circumference of the polyps and adenomas,measurement data and other informational icons for augmented displays,and control signals for semi-autonomous and autonomous operation of theendoscopic system 22.

FIG. 18 illustrates an exemplary embodiment of the colonoscopy system 10d that includes at least one illuminator 18, and at least one infraredcamera 20 and at least one RGB camera 20 c. Generally, the opticalscanning system 12 d includes the illuminator 18 providing structuredlight in combination with at least one camera 20 (e.g., high resolutionNIR cameras), and one RGB camera 20 c to produce image data forprocessing by the control system 16. The illuminator 18 may also includeat least one camera optics system 29 (not shown). The optical scanningsystem 12 d may also include the control system 26 (not shown). Theoptical scanning system 12 d may be configured such that the RGB camera20 c may be sensitive to RGB (visible) light which records a visiblelight image of the tissue under investigation using light emitted by theilluminator 18 and/or light system within the endoscope.

The RGB camera 20 c may be configured to capture visible light images ofthe tissue under investigation using visible light generated by one ormore visible light source present in all endoscopic systems 22. The RGBcamera 20 c may be selected and/or configured to possess the ability toproduce full-color, high-resolution images of the tissue underinvestigation and have both the length and width dimensions thatminimize the dimensions of the overall optical scanning system. Toproduce a sufficiently high-resolution image, an exemplary RGB camera 20c may employ a sensor with a pixel size between 1 μm and 1.4 μm and anarray of at least 1900×1900 pixels. The pixel size determines the focallength of the optical scanning system 12 d in pixels, according to theequation:

$\begin{matrix}{f_{p} = \frac{f_{mm}}{p_{mm}}} & \left( {{EQ}.\mspace{14mu} 10} \right)\end{matrix}$

where f_(p) is the focal length in pixels, f_(mm) is the focal length ofthe optical scanning system 12 d in millimeters, and p_(mm) is the pixelsize in millimeters. Sensor dimensions may not exceed 4.5 mm×3.5 mm toallow sufficient space for the infrared optical scanning components toachieve sub-millimeter or millimeter accuracy from the triangulationalgorithms implemented in the software processing system.

One or more camera optics system 29 (not shown) may be positionedadjacent to the RGB camera 20 c to facilitate light collection and toprovide the proper field of view to capture images from the entiresurface addressed by the optical scanning system 12 d. As with theilluminator 18, the camera optics system 29 may include, but is notlimited to, a single convex lens or a combination of two or more convexand/or concave lenses.

The geometrical arrangement of the illuminator 18, the high-resolutioninfrared cameras 20 (e.g., high-resolution infrared cameras), and theRGB cameras 20 c may determine the size and resolution of the opticalscanning system 12 d, with the resolution impacting the measurementaccuracy achieved after the processing software produces the threedimensional point cloud. As illustrated in FIG. 18, the high resolutioninfrared cameras 20 may be positioned on either side of the illuminator18, and in close proximity. The placement of the RGB camera 20 c mayinclude, but is not limited to, between the cameras 20 and next to theilluminator 18, outside the infrared cameras 20 (e.g., high-resolutioninfrared cameras), and outside the baseline between the cameras 20 andthe illuminator 18, for example in an orthogonal direction to the thatof the baseline. In some embodiments, the RGB camera 20 c may bepositioned adjacent to one of the cameras 20 along the same line as thebaseline of the infrared scanning components to minimize the size of theoptical scanning system 12 d.

The geometrical arrangement of the optical scanning system 12 d mayinclude, but is not limited to, a parallel configuration and a convergedconfiguration as described in detail herein. The illuminator 18 may beconfigured to produce a structured light beam that illuminates as muchof the combined FOV of the infrared cameras 20 as possible to maximizethe area of illuminated tissue recorded. The geometrical arrangement ofthe infrared components follows that described in the parallelconfiguration and converged configurations discussed in detail herein.

FIG. 19 illustrates another exemplary embodiment of the optical scanningsystem 12 e. Generally, the optical scanning system 12 e includesoptical components to isolate the infrared scanning components from thecomponents used to capture images for performing registration betweenthree dimensional point clouds. FIG. 19 illustrates an exemplaryphysical arrangement of the optical scanning system 12 e wherein thehigh-resolution infrared cameras 20 further contain an opticalwavelength filter that passes wavelengths above and including 900 nm.The illuminator 18 a may be configured to operate at a wavelength thatexceeds 900 nm, for example operating at a wavelength of 940 nm. Thecamera optics system 29 (not shown) associated with the high-resolutionRGB camera 20 c may be configured to include a wavelength filter thatpasses only wavelengths below 900 nm, such that the high-resolution RGBcamera 20 c does not collect light produced by the illuminator 18 a. Theoptical scanning system 12 e also includes illuminator 18 b, for examplea VCSEL-based source, which produces solid (flood) illumination at awavelength below 900 nm, for example operating at a wavelength of 840nm. The RGB camera 20 c therefore may collects images of illuminationprovided by the illuminator 18 b providing solid (flood) illumination,and the high-resolution infrared cameras 20 may collect images of thestructured-light illumination of the tissue produced by the illuminator18 a providing structured illumination. The use of a lower-wavelengthsolid (flood) illuminator 18 b may allow the optical scanning system 12e to take advantage of the higher response from the blood vessels atinfrared wavelengths without interference, thus providing higher qualityimage data to the mapping, registration and modeling component of thesoftware subsystem.

The properties of the components within the illuminator 18, cameras 20and 20 a, camera optics system 29 (not shown), and/or optical imagingsystem 28 (not shown), the arrangement with respect to each other andthe configuration of the complete optical scanning system 12 d withinthe mechanical housing encasing the components determine the performanceof the system in terms of lateral and depth resolution, the depth oftissue for which the target resolution is achieved, and the field ofview over which the optical scanning system 12 d can make measurements.

Generally, the RGB camera 20 c captures a visible-light image in thesame time frame that the cameras 20 capture infrared images. Specializedanalysis software employs triangulation algorithms to associatecomponents of the structured-light pattern recorded in the image of thecamera 20 with the corresponding point in the original projected patternand constructs three dimensional point clouds of the tissue illuminatedby the illuminator 18 providing structured light. The software furtheremploys algorithms for matching the patterns of blood vessels recordedby the camera 20 in successive frames in response to the solidillumination source to perform registration and stitch together thethree dimensional point clouds to produce three dimensional mappings ofthe tissue illuminated by the illuminator 18. Additional algorithmsutilize the three dimensional point clouds and three dimensional mappingto produce outputs that include, but are not limited to, detection andidentification of polyps and adenomas, measurements of height andcircumference of the polyps and adenomas, measurement data and otherinformational icons for augmented displays, and control signals forsemi-autonomous and autonomous operation of the endoscopic system.

After producing the three dimensional point cloud of one segment of theilluminated colon tissue, in some embodiments, the colonoscopy system 10d can be used to perform the colonoscopy with the physician moving theoptic hardware through the colon and performing a series of independentscans of successive colon segments. The software constructs a threedimensional point cloud representation for each segment and thenexecutes processing algorithms for detecting and measuring polyps andadenomas within each scanned segment. In some embodiments, the softwaredoes not attempt to create a three dimensional model of the tissue anddoes not attempt to stitch together the models or point clouds of eachindividual segment scanned or to locate each segment withinthree-dimensional space. As such, no additional sensors to providefeedback regarding the positioning and location of the optical scanningsystem 12 d or 12 e may be needed within the colon. In some embodiments,three-dimensional point cloud data collected from the scanning imagesmay be used to provide control signaling needed to implement autonomousor semi-autonomous operation of the endoscope employed in thecolonoscopy procedure.

In some embodiments, the physician performing the colonoscopy moves theoptical hardware through the colon with the intention of mapping all orpart of the colon. The software collects the additional data provided bythe unpatterned frames 40 or unpatterned component of composite frames40, along with data via the AHRS unit 14, in order to stitch togetherindividual images and three dimensional point clouds into a cohesive andaccurate model of the colon segment(s) of interest. A minimum level ofspatial overlap may exist between images recorded at successive timeintervals as the optical scanning system 12 d or 12 e moves through thecolon to allow mapping and registration functions using the unpatternedframes 40 or unpatterned components of composite frames 40. In someembodiments, the optical scanning system 12 d or 12 e may also employadditional sensors that may include, but are not limited to, othersensors integrated with the optical scanning system 12 d or 12 e, sensorarrays within the endoscope 22 used in the colonoscopy procedureindependent from the optical scanning system 12 d or 12 e, and/or sensorarrays located externally to the colon or patient that detect signalswhich allow the software to locate the position of the optical scanningsystem 12 d or 12 e within the patient. The additional data provided bythe optical scanning system 12 d or 12 e and the cohesive threedimensional mapping of the colon may provide sufficient information togenerate signaling needed to implement autonomous or semi-autonomousoperation of the endoscope 22, especially in the colon where theendoscope 22 may need to change shape in the areas behind the scanninghead, and support the detection and mitigation of loops in the colonduring the colonoscopy procedure.

Referring again to FIG. 1, therein is a block diagram an exemplarycontrol system 16 for use in any of the colonoscopy systems 10-10 e andversions thereof. The control system 16 is able to embody and/or executethe logic of the processes described herein. Logic embodied in the formof software instructions and/or firmware may be executed on anyappropriate hardware. For example, logic embodied in the form ofsoftware instructions and/or firmware may be executed on dedicatedsystem or systems, on distributed processing computer systems, and/orthe like. In some embodiments, the logic may be implemented in astand-alone environment operating on a single system and/or logic may beimplemented in a networked environment such as a distributed systemusing multiple computers and/or processors. For example, microprocessors90 of the control system 16 may work together or independently toexecute processor executable code using one or more memories 92.

FIGS. 20-35 illustrate methods of using the colonoscopy systems 10-10 ein further detail. Generally, software within the control system 16 canutilize measurement and imagery data to assist the operator in detectingthe existence of a polyp or adenoma within the patient's colon and inmeasuring the size and dimensions of the polyp, both of which provecritical in identifying and mitigating problems that can later lead tocolorectal cancer. If the operator decides to cut a polyp during thecolonoscopy procedure, the software can provide imaging data todetermine which tool the operator should use to cut the polyp and toprovide measurements and positioning data to properly guide and positionthe tool to ensure the operator removes the entire polyp. At higherlevels, software within the control system 16 can utilize the mappingand registration data and modeling functions to assist the operator inidentifying the formation of loops within the colon, provide threedimensional mapping of the patient's colon that the operator or thepatient's doctor can use to track changes in health over time, and toprovide extensive documentation of the entire colonoscopy procedure forlater evaluation and analysis. The software, using the mapping andmodeling functions to pinpoint the location of the cancerous tissuewithin the colon in a manner that allows the surgeon to rapidly locateand remove the tissue. The detailed information produced by the softwaresystems allows for the generation of control signals to guide theendoscope 22 through the procedure in either an autonomous orsemi-autonomous manner, or to generate augmented displays which assistthe operator in guiding current endoscopic tools during a procedure. Theautonomous, semi-autonomous, and augmented operating modes may allow,for example, surgeons and other specialists not specifically trained ingastroenterology to perform colonoscopy procedures at a high level andmeet the increasing demand for using colonoscopy for both preventativeand treatment purposes.

Regarding polyp detection and measurement, the control system 16 maydetermine a three-dimensional point cloud by extracting from the imagedata provided by the optical scanning system 12. The three-dimensionalpoint cloud may be used to further: (1) detect the presence of polypsalong the wall of the colon; and/or (2) to make measurements on the sizeof the polyp, including but not limited to the height and thecircumference of the polyp. It should be noted that for the opticalscanning system 12 d and 12 e illustrated in FIGS. 18 and 19respectively, the RGB image provided by the RGB camera 20 c may providean additional method that uses the three dimensional point cloud and adeep-learning engine to detect and measure polyps.

Polyp Detection

Polyp detection may be based on the output from the optical scanningsystem 12 and utilizes a geometrical analysis method. Generally, thereare three operations: modeling, model fitting, and polyp identification.

In the modeling operation, the control system 16 for polyp detection andmeasurement creates a model of the three-dimensional surface (the set ofpoints in the three dimensional point cloud) by approximating thesurface using a set of equations. In the modeling operation, the controlsystem 16 attempts to create a large area model that accuratelyapproximates the geography of the surface over a large scale, instead ofmodeling accurately every minor variation in the surface structure. Byapproximating the large scale geometry, the model does not include thevariations that correspond to the presence of a polyp along the surface,and this omission in the model may provide means to extract the polypfrom the background tissue in the image of the colon wall.

The set of equations used to construct the mathematical model for themodeling operation can include, but is not limited to, a Bezier surface,a splines surface, a surface described by a non-uniform rational Bspline (NURBS), and the like. Each mathematical model exhibits strengthsand weakness in accurately modeling different types of surfaces, and theprocess of choosing the best mathematical model for differentapplications requires the user to test and compare the operation ofseveral mathematical models based on accuracy and successful rate offeature detection in the later functional steps. In what follows thedescription of the functional operation will assume a NURBS model as arepresentative example.

The set of equations within a chosen mathematical model contains a setof fitting parameters. The control system 16 for polyp detection andmeasurement chooses the parameters to allow the equations to bestapproximate the three dimensional surface through a fitting process.Possible methods for implementing the fitting process include, but arenot limited to, an iterative method and a robust method.

In the iterative method, the control system 16 selects an initial set ofvalues for the fitting parameters and measures the accuracy of theapproximate surface with respect to the actual surface by methods thatinclude, but are not limited to, average mean square error. The controlsystem 16 may then vary the fitting parameters to minimize the accuracymeasurement (for example, minimizing the mean square error). In theiterative method, the number of iterations may be limited, defined asthe number of times the method varies the fitting parameters and teststhe accuracy, to a small amount to avoid the algorithm from accidentallyfitting the surface to a polyp the colonoscopy system 10 wants todetect. Including the polyp in the surface prevents the control system16 from differentiating the polyp from the background tissue.

An alternative to the iterative method is a robust method. The robustmethod may be used when the three dimensional point cloud contains anumber of different structures and/or when the three dimensional pointcloud contains noise, for example. Such methods include, but are notlimited to, the Random Sample Consensus (RANSAC) method. The robustmethod commonly performs good fitting of the large trends in theavailable data and tends to ignore points that are outliers with respectto the larger trend. For detecting polyps, the outlier points representthe polyps, and therefore exclusion from the fit of the larger surfaceenhances the performance of the overall polyp detection algorithm inidentifying the location of the polyps.

As an example, FIG. 20A shows a three-dimensional point cloud 100obtained from the optical scanning system 12 (not shown) when scanning atissue model with simulated polyps. FIG. 20B shows the outcome offitting the tissue surface using the combination of a NURBS model andthe iterative fitting method. In both FIGS. 19(a) and 19(b), the 3Dpoint cloud is colored in yellow. In FIG. 19(b), the fitted surfacemodel is colored in green.

Polyps may be detected via distance map computation and polypextraction. During distance map computation, the control system 16creates a spatial map of the distance (or difference in height) betweenthe fitted model and the original three dimensional point cloud. As anexample, one method of computing the distance map begins by findingtwenty points nearest to a point of interest P. Using only those twentypoints, an approximate plane is constructed using a fitting model suchas NURBS, and distance is determined between the surface of the planeand the point P along a direction normal to the surface of the plane.For each lateral position along the surface, described by a coordinatepair (x,y), the method stores the computed distance in a matrix. Oncethe method determines the distance for all points P in the point cloud,a map (or plot) is generated of the computed distance as a function oflocation. Regions with small values of distance are regions wherein thefitted surface closely approximates the three-dimensional point cloud,and therefore, those regions do not likely contain polyps. Regions withlarge compute distance represent likely locations in which polyp(s) mayexist. For example, FIG. 21 illustrates a computed distance map 110wherein the regions with higher intensity indicate larger computeddistances, and therefore indicate a high likelihood that a polyp existswithin the regions.

During polyp extraction, the computed distance map 110 in FIG. 21 may beprocessed to determine the position of possible polyps and to classifythe possible polyps as either true polyps that the control system 16will report or false positives that the control system 16 will rejectand not report to the operator. In some embodiments, the data in FIG. 21passes through a process wherein the continuous data is converted intobinary data, with regions over a certain distance labeled as possiblepolyp sites and other areas labeled as not likely to contain a polyp.These two labels are shown as white pixels and black pixels respectivelyin FIG. 22A.

After completing the binary processing, the binary image passes througha filtering stage. In the filtering stage, the control system 16computes the mean intensity, which is proportional to the mean distance,of the region around each possible positive. Regions that produce a meanintensity below a pre-determined threshold intensity are rejected by thefilter, leaving only the most likely candidates for polyps for furtherprocessing and identification, as shown in FIG. 22B. Training of thecontrol system 16 by an operator or through other means (e.g.,artificial intelligence), including but not limited to neural networkalgorithms, provides the threshold value that maximizes the accuracy ofthe control system 16 in correctly ignoring false positives. The regionsremaining after filtering are then projected back onto the originalthree dimensional point cloud for further processing, as shown in FIG.23.

In some embodiments, the next processing step includes, but is notlimited to, input of the data in FIG. 23 into a pre-trained artificialintelligence (AI) engine to complete the process of detecting andlocalizing the polyps. AI engines may include, but are not limited to,supervised AI engines such as a meta heuristic AI engine or an edgekey-points detection engine and unsupervised AI engines.

In using edge key-points detection, key points on the depth map areselected that are repeatable (e.g., with respect to noise and localvariations) and distinctive (e.g., the area around the keypoint has aunique shape or appearance that a feature extractor can capture). Ingeneral, more keypoints used in the algorithm increases the accuracy ofpolyp detection, and may also increase the processing time required bythe algorithm. Proper keypoint selection balances the need for detectionaccuracy and real-time feedback to the operating physician. Severalkeypoint algorithms exist in the literature. Proper selection of analgorithm requires experimental verification and testing for a specificapplication. Each keypoint is described by a vector of values calculatedfrom different feature parameters targeted by the algorithm. The featureparameters selected for the control system 16 for polyp detection andmeasurement may reflect unique characteristics of the polys compared tonominal features (or lack thereof) of the tissue in the colon wall. Thecontrol system 16 computes the feature values and uses a set of weightsto classify the keypoint as either part of a polyp or part of theregular surface of the colon wall. The weights are obtained by applyingand training a two-class classifier, which include, but are not limitedto linear classifiers, decision trees, random forest, neural networksand nearest neighbor. Training occurs on a pre-constructed sample set ofdata configured to contain all of the cases the trainer expects theclassifier to encounter. The training occurs during the originalconstruction of the control system 16, and remains fixed within thesoftware after full system development. An example of the keypointdetection process output is shown in FIG. 24A. All keypoints identifiedas belonging to a polyp that form an independent cluster are groupedtogether and labeled as a single polyp, as shown in FIG. 24B. The pointsallow a second algorithm, including but not limited to an active contouralgorithm, to estimate the polyp border, as shown in FIG. 24C.

In using the meta heuristic AI engine (i.e., deep learning AI engine),all of the sub-processes described in the edge key points detection aremerged into a single module that performs the entire polyp detectionprocess. This engine requires more training samples and higherprocessing power to train than the keypoints engine.

In using the unsupervised AI method, training is not required. Themethod divides the three dimensional point cloud into segments based ongeometrical distance, and classifies each segment into a polyp ornon-polyp category based on geometrical distance. The segmentationprocess occurs unsupervised, meaning that the process is notobject-aware—the process does not know ahead of time what objects it islooking for. Segmentation algorithms include, but are not limited to,density-based clustering, mean-shift clustering, watershed segmentation,normalized graph cuts, and the like. The process requires tuning ofseveral parameters, of which the most important is the distancefunction, which describes the distance between each pair of points inthe three dimensional point cloud data. Polyp classification uses thegeometrical properties of polyps with respect to the background tissueto classify whether a segment belongs to a polyp or to the backgroundtissue. The classification process can occur in an unsupervised orsupervised manner. Post processing may be required to combine segmentsclassified as belonging to a polyp into a single polyp when the segmentsare directly adjacent to each other. Post processing algorithms include,but are not limited to, active contour format algorithms.

For the colonoscopy systems 10-10 d, the software of the control system16 implements deep learning and artificial intelligence to enhance theaccuracy of the results obtained by the colonoscopy system 10-10 d. Thesoftware combines data from the three dimensional point clouds producedby the optical scanning system 12 and/or three-dimensional mapping fromthe registration and modeling systems with the visible light dataobtained by the RGB camera 20 c that the existing endoscopic system 22uses to provide the physician with images of the surgical procedure. Thedeep learning and artificial intelligence algorithms reconcile the threedimensional data with the visible light imagery and combine them into anenhanced image that provides additional detail and clear referencepoints for the location of polyps, adenomas, and other features ofinterest.

Referring to FIG. 25, for the colonoscopy system 10 d or 10 e, whereinthe RGB camera 20 c provides an RGB image, the software implements adeep learning artificial intelligence algorithm to facilitate detectionand measurement of polyps, either as a complement to or in lieu of thedescribed method using the output from the infrared scanner only. Forthe colonoscopy system 10 d or 10 e, the RGB image and the depth map(i.e., three dimensional point cloud) act as inputs to a polyp detectionblock that employs deep learning artificial intelligence.

The polyp detector block 111, as shown in FIG. 25 contains a first layerthat employs two image processing subnetworks 112 and 114, each tunedwith different weights to process the information contained in the depthmap 118 and the RGB image 116. A fusion layer 120 works to combine thedata obtained from the output of each subnet 112 and 114 into a singleset that correlates/maps each component of the depth map 118 to a pixelin the RGB image 116. An artificial intelligence network 122, which caninclude any of currently known and accepted models and learningtechniques, processes the fused data to extract features such as polyps.The artificial intelligence network may be supervised or unsupervised,as described previously.

Poly Measurement

Polyp measurement utilizes output from the detection of polyps via thecontrol system 16 and/or underlying data from the three dimensionalpoint cloud to perform measurements on the polyp. The measurementsinclude, but are not limited to, perimeter and surface area, forexample.

The control system 16 may determine the perimeter of the polyp as thesum of distances between successive points along the border of the polypdetermined by the polyp detection function, such as that in FIG. 24C.FIG. 26 shows a three dimensional point cloud 130 with the data points132 located along the polyp border. The control system 16 may uselengths L_(P1), L_(P2) . . . L_(PN) of lines 134 determined between thedata points 132 and sum such lengths to calculate the perimeter of thepolyp.

In some embodiments, the control system 16 may use one or more curvefitting algorithms to achieve better accuracy in calculating theperimeter. FIG. 27A shows an original shape of a polyp 140 and the datapoints 142 from a three dimensional point cloud that lie along theoriginal shape. Without knowing the original shape, the control system16 may estimate distances between the data points using straight lines,as shown in FIG. 27B. The straight line approximation reduces theaccuracy of the perimeter calculation as a result. Using one of severalavailable curve-fitting algorithms, control system 16 can add curvatureto the lines between the points, as shown in FIG. 27C, resulting in amore natural shape that better matches the original shape. When thepolyp measurement function calculates the lengths of the curved lines, abetter approximation of the perimeter may be achieved.

In some embodiments, the control system 16 may determine surface area ofthe polyp. To compute the surface area of the polyp, the control system16 may collect most or all of the three dimensional point cloud datapoints 132 that the polyp detection function assigned to a polyp. Thecontrol system 16 may form a surface mesh by constructing triangles 136between sets of three data points 132 along the surface of the threedimensional point cloud data 130, as shown in FIG. 28B. The controlsystem 16 may determine an area of each triangle within the mesh. Thecontrol system 16 may determine total surface area by summing the areasof the individual triangles 136.

Referring to FIG. 1, in some embodiments, the control system 16 mayprovide mapping, registration and modeling during a colonoscopyprocedure. Generally, the control system 16 may collect and/or processdata from the unstructured (solid) image data and utilize mapping andregistration algorithms to align and stitch together three dimensionalpoint clouds into a cohesive, accurate three dimensional model ofscanned tissue. The control system 16 may also collect and/or processdata from the AHRS unit 14 to provide additional input to the mappingand registration algorithms. The resulting three-dimensional model mayprovide detection of colon loops, accurate location of features ofinterest within the length of the colon, construction of baseline modelsand data of the patient's health, and support semi-autonomous andautonomous operation of the endoscope during a colonoscopy, for example.

For mapping, also referred to as localization, the control system 16determines position and attitude (orientation) of the optical scanningsystem 12 at each time an image frame is captured by the camera(s) 20and/or 20 a. For registration, the control system 16 determines thechange in position and the change in attitude experienced by the opticalscanning system 12 during the time between two images captured by thecamera(s) 20 and/or 20 a. For mapping, the control system 16, therefore,determines location in space (x, y, z) and angular orientation (θ, φ, ψ)of the optical scanning system 12 at a particular time t_(i). Forregistration, the control system 16 may determine difference in location(Δx, Δy, Δz) and the difference in orientation (Δθ, Δφ, Δψ) of theoptical scanning system 12 between the original time t_(i) and a latertime t_(i+1). To accomplish the objective of the control system 16 formapping and registration, the control system 16 may (a) identifyfeatures within the colon that facilitate measurement of the differencesin location and orientation and (b) associate captured three dimensionalpoint clouds with specific locations and orientations of the opticalscanning system 12.

Identifying Features within the Colon

Generally, there exists a short operating distance between the opticalscanning system 12 and the colon wall. Additionally, the surface of thewall tissue may be relatively smooth and featureless. Such features mayprohibit use of methods that capture images of a large area around thearea targeted for scanning which can overlap significantly between twoframes captured at times time t_(i) and t_(i+1). Additionally, therelatively smooth and featureless surface of the colon wall may providefew opportunities the control system 16 to place markers on clearfeatures that the control system 16 can identify easily in two or moreframes.

The colonoscopy system 10 may overcome registration difficulties bycapturing and matching patterns of blood vessels within or on the colonwall. To accurately capture and match the patterns of blood vessels, theoptical scanning system 12 project an illumination on the wallconsisting of more than simply structured light patterns. For the mostaccurate recovery of the blood vessel patterns, the colonoscopy system10 may record images that, in whole or in part (composite), containsolid (unpatterned) illumination of the colon wall.

As described in further detail herein in relation to FIGS. 7-17,illumination and capture of images containing solid (unpatterned)illumination of the colon wall may be provided by the illuminator 18providing an infrared light source and/or the illuminator 18 providingsolid illumination. Once the camera(s) 20 capture images of the colonwall illuminated by unpatterned (solid) light, registration between twosuccessive images can be performed by methods that include, but are notlimited to, general purpose registration algorithms, algorithm(s) basedon generation and matching of binary images, and/or the like.

In some embodiments, general purpose registration algorithms may be usedto register data between two successive images as shown in FIG. 29. Thegeneral purpose registration algorithms may operate by matching keypoints that appear in both of the images. The algorithm identifiesunique features or points in the original image or frame, and thensearches for the same feature or point in the next image or frame.

In some embodiments, an algorithm based on the generation and matchingof binary images may be used to register data between two successiveimages as shown in FIGS. 30A and 30B. FIG. 30A illustrates a capturedimage of blood vessels under unpatterned (solid) illumination. Since theblood vessels represent a unique structure on the colon wall,binarization of the captured image, as shown in FIG. 30B, produces aunique binary (black and white) pattern that simplifies the registrationprocess and makes the process more robust. Modeling of the blood vesselsmay include, but is not limited to, using a NURBS curve, instead ofpoint-to-point matching, which provides more robust matching thanpoint-to-point matching, for example.

As described in further detail with regard to FIGS. 18 and 19, thecolonoscopy system 10 may use the RGB camera 20 c to capture full-color,visible light images of the tissue under illumination from theunpatterned visible light source present in all endoscopic systems. Oncethe RGB camera 20 c captures images of the colon wall illuminated byunpatterned (solid) visible light, registration between two successiveimages can be performed by methods that include, but are not limitedgeneral purpose registration algorithms, an algorithm based ongeneration and matching of binary images similar to that which isdescribed in between above, and/or the like. For example, in someembodiments, general purpose registration algorithms may be used forregistration between images and operate by matching key points 140 a,140 b . . . 140 n that appear in both of the images, as shown in FIGS.31A and 31B. The algorithm identifies unique features or points in theoriginal image or frame, and then searches for the same feature or pointin the next image or frame. For the visible light images when using theRGB camera 20 c, for example, the image may require post-processing toensure that the blood vessels are sufficiently well-defined within theimage for the key-point detector to operate effectively. For example,FIG. 32A shows a raw image of the colon wall collected by the RGB camera20 c, in which the blood vessel edges are ill-defined, providing veryfew features for the key-point detector to detect. Any of several commonimage processing techniques can produce the image in FIG. 32B, whereinthe edges and patterns of the blood vessels appear quite clearly,allowing ready and more accurate extraction of the key points.

Mapping and Registration Between Point Clouds

Mapping and registration may be based on alternating images and mappingand registration based on hybrid images in relation to FIGS. 7-17. Inmapping based on alternating images, the optical scanning system 12 mayalternately capture images of the colon wall for structure-light(patterned) illumination and unpatterned (solid) illumination. Forexample, the optical scanning system 12 may illuminate the colon wallwith unpatterned (solid) illumination in odd time frames (t₁, t₃, t₅)and illuminate the colon wall with a structured-light pattern in eventime frames (t₂, t₄, t₆). During odd time frames, the optical scanningsystem 12 may capture images of the blood vessel pattern, binarize theimages, and perform matching between the two images. The matchingprocess allows the software to estimate the change in position andorientation that occurred in the time between the two frames. During theeven time frames, the optical scanning system 12 may capture thestructured-light images and the control system 16 may determine thethree dimensional point clouds for the colon wall. The control system 16may estimate the position and orientation of the optical scanning system12 for the even frames using methods that include, but are not limitedto (a) assuming smooth motion of the optical scanning system 12 duringthe time interval between the odd frames on either side of the evenframe, (b) using state-of-the-art point cloud registration algorithmsbetween two successive even frame images, and/or (c) combining dataobtained from the odd frames with data obtained via the AHRS unit 14 atevery frame (both even and odd). Position and orientation data estimatedfrom the odd frames may be considered to be highly reliable on oddframes and approximate on even frames. Position and orientation dataestimated from the AHRS unit 14 may provide only rough accuracy. Theaccuracy of position and orientation data estimated from the even framesmay depend on the number and uniqueness of geometrical features withinthe captured images.

Referring to FIG. 33, at Frame 1, the optical scanning system 12 maycapture blood vessel images under unpatterned (solid) illumination andorientation data from the AHRS unit 14. At Frame 2, the optical scanningsystem 12 captures images of the colon wall under structured-lightillumination as well as orientation data from the AHRS unit 14, and thecontrol system 16 determines the three dimensional point cloud for theframe. At Frame 3, the optical scanning system 12 captures blood vesselimages under unpatterned (solid) illumination and orientation data fromthe AHRS unit 14, and the control system 16 determines three dimensionalbetween Frames 1 and 3 using the blood vessel markers in Frames 1 and 3and the orientation data from the AHRS unit 14 from Frames 2 and 3. AtFrame 4, the optical scanning system 12 captures images of the colonwall under structured-light illumination, captures orientation data fromthe AHRS unit 14, and the control system 16 determines the threedimensional point cloud for the frame, and performs registration betweenFrames 2 and 4 using point clouds from Frames 2 and 4 and the AHRSorientation data from Frames 3 and 4. This process may be repeatable forall following frames.

Registration can occur on either a global or semi-global level. Forglobal registration, small errors that accumulate over time may bemitigated by recording images when moving the endoscope 22 into thecolon and when moving the endoscope 22 out of the colon. Using imagescollected on both the inward and outward trips creates a loop thatallows elimination or mitigation of the errors. For semi-globalregistration, registration may be applied to small groups of N sequencedframes that all share some minimal level of mutual information(overlap), as shown in FIG. 34. The use of multiple, semi-globalregistrations may allow for corrections that ensure better accuracy andless drift in the colonoscopy system 10 over time.

In some embodiments, mapping and registration may be based on hybridimages. For example, the optical scanning system 12 may illuminate thecolon wall with a composite or hybrid image containing areas withstructured-light illumination and areas with unpatterned (solid)illumination, such as, for example, in FIGS. 6A-6C. The system uses theareas of unpatterned (solid) illumination to capture and performmatching on images of the blood vessel patterns in the colon wall. Thecontrol system 16 may use the areas of structured-light illumination toconstruct one or more three dimensional point clouds of the colon wall.

As an example, at Frame 1, the optical scanning system 12 captures animage of the colon wall illuminated by the hybrid or composite patternfrom the illuminator 18 and captures orientation data from the AHRS unit14. Also at Frame 1, the control system 16 may construct a threedimensional point cloud for the area illuminated by the structured-lightpattern and binarizes the blood vessel images collected from the areasilluminated by the unpatterned (solid) illumination. At Frame 2, theoptical scanning system 12 captures an image of the colon wall aftermoving along the colon, captures orientation data from the AHRS unit 14,the control system 16 may construct the point cloud for the areailluminated by the structured-light pattern, and binarizes the bloodvessel images collected from the area illuminated by the unpatterned(solid) illumination. The control system 16 may perform two matchingprocesses—one matching process based on the three dimensional pointclouds and a second matching based on matching of the binarized bloodvessel images. The control system 16 may use results of both matchingprocesses to perform registration between the frames. The process mayperform registration between Frames 2 and 3, Frames 3 and 4, and soforth in the same manner.

Mapping and registration for embodiments that include the RGB camera 20c (e.g., embodiments described in relation to FIGS. 18 and 19), mayinclude extraction of three dimensional coordinates, matching betweenimages collected in two consecutive frames, determination of atransformation matrix between matched sets to determine rotation andtranslation occurring between frames, and global registration.

Calibration data for the RGB camera 20 c may allow the control system 16to determine a transformation matrix between the RGB camera 20 c and thestructured light infrared camera 20. This transformation matrix mayallow the control system 16 to accurately transform the depth map orthree dimensional point cloud produced by the infrared scanningcomponents into the frame of reference of the RGB camera 20 c, and viceversa, to facilitate mapping. Using the transformation matrix and thedepth map, the control system 16 can calculate the three dimensionalcoordinate [X, Y, Z] of each pixel of the RGB image, and thus cancompute the three dimensional coordinate of key points or NURBS curvesextracted from the RGB image.

The control system 16 may perform matching between images collected intwo consecutive time frames. For the key-point detection approach, thecontrol system 16 may perform matching based on the spatial featurescontained within the window around the key point. Matching algorithmsuseful for this process may include, but are not limited to, ORB, SIFT,SURF, and/or KLT. For example, the algorithm for use with consecutiveframes may be KLT, as the algorithm proves robust for a consecutiveframe approach. For the NURBS curve approach, the control system 16performs matching between curves. The control system 16 may find atransformation matrix that minimizes the differences between theparameters of the two curves. If such a transformation matrix is foundand satisfies the rigidity condition (for example, no changes in thevessel structure occur using the transformation matrix), the two curvesare matched.

Once a set of matched key points is identified, the control system 16may determine the transformation matrix between the matched sets todetermine the rotation and translation that occurred between the initialframe and the successive frame. The transformation matrix T/+¹ iscomputed using the equation:

$\begin{matrix}{\begin{bmatrix}X_{1} & X_{2} & X_{3} & \; \\Y_{1} & Y_{2} & Y_{3} & \ldots \\Z_{1} & Z_{2} & Z_{3} & \;\end{bmatrix} = {T_{i}^{i + 1}\begin{bmatrix}X_{1}^{\prime} & X_{2}^{\prime} & X_{3}^{\prime} & \; \\Y_{1}^{\prime} & Y_{2}^{\prime} & Y_{3}^{\prime} & \ldots \\Z_{1}^{\prime} & Z_{2}^{\prime} & Z_{3}^{\prime} & \;\end{bmatrix}}} & \left( {{EQ}.\mspace{14mu} 11} \right)\end{matrix}$

wherein [X_(i), Y_(i), Z_(i)] are the coordinates of the matched keypoints in the initial frame and [X′_(i), Y′_(i), Z′_(i)] are thecoordinates of the matched key points in the successive frame. Methodsfor computing the transformation matrix include, but are not limited to,LSE or other linear solving algorithm. To improve the accuracy of thetransformation matrix, the computation may also employ algorithms thatremove points within the data sets that were incorrectly matched duringthe matching process. For example, two points representing distinct ordifferent features may be incorrectly identified as representing thesame feature and thus removed. Possible solutions include, but are notlimited to, robust estimators such as RANSAC, which can greatly improveaccuracy of the calculated transformation matrix even if only 40% of thematches produced in the previous step represent valid matched.

For the NURBS curve approach, the process of identifying matches betweena pair of curves may produce a transformation matrix, and therefore, aseparate transformation matrix determination may not be required. Inpractice, the process of matching multiple curves between twoconsecutive frames may result in the determination of multipletransformation matrices. The multiple transformation matrices may not beidentical in some or all of the terms. Methods for producing a singletransformation matrix for two consecutive frames may include, but arenot limited to, averaging the multiple matrices and performing globalmatching of all the curves simultaneously rather than individually, forexample.

To correct for accumulated error in the registration of consecutiveframes over the length of the colon, the control system 16 may perform aglobal registration process. The control system 16 may periodicallylabel a time frame as a reference frame. For example, the control system16 may label every 30th frame as a reference frame. The control system16 stores the key points and descriptor data related to the key pointsover the entire travel of the endoscope 22. The control system 16 maythen performs global registration and/or semi-global registration in amanner similar to that described in FIGS. 33 and 34. FIGS. 35A and 35Billustrate the difference in the computed paths of the endoscope beforeand after the process of global registration respectively. In FIG. 35A,the line 150 is the estimated path and the line 152 is the true pathtaken by the endoscope 22. FIG. 35B demonstrates the accuracy of theestimated path 154 attained using global registration methods.

During use the colonoscopy system 10 may provide an operator (e.g.,physician) performing a colonoscopy procedure with valuable detection,measurement and analysis tools that enhance the accuracy andeffectiveness of the procedure. For example, the colonoscopy system 10may construct one or more three dimensional point clouds and/or threedimensional models of tissue within the colon, which can contribute toaugmented artificial intelligence guidance for the endoscopic operatorand further applications in simulation of the procedure for instructionand training. In some embodiments, the colonoscopy system 10 may provideone or more measurements of key features within the scanned volume,including lateral, perimeter, and depth measurements of polyps andadenomas. Knowledge of these parameters may allow an operation (e.g.,physician) performing the colonoscopy to make informed decisions (e.g.,removal of the polyp or adenoma, selection of proper tool for removal,etc.).

In some embodiments, the colonoscopy system 10 can construct one or morethree dimensional models of one or more areas of interest, as well asanalyze raw data and the resulting three dimensional point clouds todetect features and/or issues with one or more objects that an operatormay miss. For example, the operator may miss a polyp within the colonduring a colonoscopy because the polyp is small, visually blends intothe background, or is obscured by folds or flaps of tissue in or on thewall of the colon.

In some embodiments, the colonoscopy system 10 may produce threedimensional point clouds and/or three dimensional mapping of a colonwall that can provide image data and/or measurement input to thesoftware processing components to generate some form of alarm or warningsystem. The alarm or warning system may notify an operator (e.g.,physician) when the endoscope approaches too closely to the colon wall,and/or when the orientation, angle, and/or placement of the endoscopewithin the patient is incorrect. Such an alarm or feedback to theoperator may prevent accidental injury to the patient that couldpotentially occur during the procedure.

The colonoscopy system 10 may provide imagery and/or measurement data ofsufficient precision and detail to support development of endoscopicsystems with limited to full autonomous operating capabilities. Theoptical scanning system 12 mounts on or integrates within a head of anendoscope, and may utilize one or more proximity sensors in addition tohardware within the optical scanning system 12. The optical scanningsystem 12 may be mounted or located in such a way to provide threedimensional point clouds, mapping and registration data, and measurementdata for the forward direction and/or along the sides of the head of theendoscope. The control system 16 may utilize the resulting point clouds,mapping and registration data, and measurement data to operate atdifferent levels of autonomy during the colonoscopy procedure. In asemi-autonomous level of operation, the operator may retain primarycontrol of the endoscope, and the control system 16 may provide alarmsin some form to warn of potential dangers or errors that could lead toinjury to the patient, temporarily take control of the endoscope toavoid such dangers, and provide imagery or other forms of feedback suchas augmenting existing displays with relevant data and/or icons, thatassists the operator during the task. In a fully autonomous level ofoperation, the colonoscopy system 10 may exert primary control over theendoscope, using the three dimensional point cloud and measurement dataas inputs to control and feedback systems that utilize artificialintelligence techniques to direct the movements of the endoscopicinstrument autonomously, within a set of parameters dictated by theoperator or the procedure. The operator may retain the ability tooverride and/or pause the autonomous operation to inspect something ofinterest to the operator or to perform some operation outside of themain procedure, and then allow the autonomous operation to continue.

For example, colonoscopy procedures require that the endoscope navigatesharp bends in the colon. In either autonomous or semi-autonomous modesof operation, the control system 16 generates a model or map of thesurrounding tissue from the three dimensional point cloud and themapping and registration data collected along the endoscope's route. Theoperator or control system 16 can utilize the map to control not onlythe head of the endoscope but also the entire body of the endoscope,ensuring that the shape of the endoscope body conforms to the paththrough the patient's colon at all times and thus minimizing the chanceof causing damage to the patient. The whole endoscope therefore movesautonomously even though the optical scanning system 12 is located onlyat the head of the endoscope.

FIGS. 36A-36G illustrate an exemplary embodiment of a colonoscopy system10 f in use. An optical scanning system 12 f of the colonoscopy system10 f may be incorporated into a colonoscopy instrument currently usedwithin the art. For example, the optical scanning system 12 f mayinclude at least one illuminator 18 capable of producing both patternedand solid illumination or combinations (e.g., NIR illumination), and atleast one camera 20 integrated directly within an endoscope 22 and theexisting endoscope component. The colonoscopy instrument may alsoinclude current components found in a colonoscopy system within the artincluding, but not limited to, a RGB camera, an existing visible lightsystem 160, an air nozzle 161, an instrument channel 162, and/or a waterjet 163.

Referring to FIG. 36B, an operator may insert the endoscope 22 into acolon 164. The visible light system 160 is configured to capture one ormore visible light images of the colon. The colonoscopy system 10 f mayprovide such visible light images to an operator in any of the commonmethods used in current colonoscopy systems.

The colonoscopy system 10 f may capture three dimensional images and/ortwo dimensional images of the colon from the NIR cameras 20 a and 20 b,driven by the illuminator 18 in patterned illumination mode and solidillumination mode, respectively. The control system 16 may use the threedimensional images to construct local three dimensional point cloudsand/or three dimensional models, for example. The control system 16 mayuse the two dimensional images to perform registration calculationsand/or stitch the local three dimensional point clouds into one or morethree dimensional models of the colon.

The control system 16 may provide output to the operator (e.g., thethree dimensional model, measurements, alerts) by one or more outputdevice 166, including, but not limited to, implementations as a anePaper, computer monitor, speaker, screen, touchscreen, television set,smart phone, PDA, cell phone display, printer, optical head-mounteddisplay, an augmented reality system, combinations thereof, and/or thelike. FIG. 36B illustrate the output device 164 as a screen having aplurality of sections 168 a-168 c for providing information and data toan operator. For example, in some embodiments, at least one section 168a of the output device 166 may provide visible light data or a separatevisualization device or screen. In at least one section 168 b, theoutput device 166 may provide a three dimensional model 170 of the colon164 based on data received by the optical scanning system 12 f. Thecontrol system 16 may update and add to the three dimensional model 170presented to the operator in real time after completing each cycle ofscanning and processing as described in detail herein.

As the endoscope 22 approaches a polyp or feature within the colon 164(shown as element 180 in the visible light section 168 a), the controlsystem 16 using data obtained by the optical scanning system 12 f maydetect presence of the feature 180 (e.g., polyp). Presence of thefeature 180 may be visual, audibly, and/or tactilely provided to theoperator. For example, in some embodiments, the operator of thecolonoscopy system 10 f may alert the operator using methods thatinclude, but are not limited to, visual indicators and/or icons on thethree dimensional model in section 168 b, augmenting the visible lightimages in section 168 a with icons or other symbols, as shown in FIG.36C.

Using the registration and model construction capabilities of the threedimensional and/or two dimensional imaging systems, the control system16 may provide the operator with accurate location, size, and featureinformation of the feature 180 to the operator. Methods for presentingdata regarding location, size, and feature information of the feature180 to the operator include, but are not limited to, augmenting thevisible image (shown in FIG. 36D), displaying the information on thethree dimensional colon model, delivering the information to a separatescreen or viewing device, and/or a combination thereof.

In some embodiments, the operator can store real time data provided bythe control system 16. Stored data may be used in application including,but not limited to, follow-up observations of the feature 180 (e.g.,polyp) during later procedures, removal of feature 180 (e.g., polypremoval) in follow-up procedures, maintaining records of patient health,and/or the like.

In practice, residual materials may remain in the colon 164 afterpreparation procedures, including food remnants and stool. The presenceof such residual materials can obscure abnormal tissues from theoperator's view, resulting in missed polyps, adenomas, or cancerousgrowths, or result in the recording of a false positive recording of anabnormal tissue. The colonoscopy system 10 f provides the operator withtools to improve correct identification of objects within the colon. Forexample, if the operator cannot discern whether the feature 180 consistsof a polyp, stool, or other residual materials, the operator can use oneor more two dimensional images provided by the optical scanning system12 f to reveal whether the object contains or obscures blood vessels.Since a polyp, adenoma, or other abnormal tissue must be fed by bloodvessels, an object containing images of blood vessels in the twodimensional image has a high probability of consisting of abnormaltissue. An object for which the two dimensional images do not indicatethe presence of blood vessels has a high probability of consisting ofstool or other residual materials, as these objects should not containblood vessels, and will block the blood vessels in the colon 164 fromthe view of the optical scanning system 12 f.

Referring to FIG. 36E, if the colon 164 contains a loop 184 (e.g., alphaloop, reverse alpha loop, N loop, spiral sigmoid loop, or other loopknown within the art), the control system 16 can detect the loop 184from the three dimensional colon model 170. The three dimensional model170 contains location information for each colon section scanned by theoptical scanning system 12 f. The control system 16 can highlight theloop 184 using methods that include, but are not limited to, changingthe color of the three dimensional model 170 in the looped segment, theuse of arrows and/or other icons. In the same manner as the detection offeatures 180, the control system 16 can present data on the exactlocation of the loop 184 to the operator by methods that include, butare not limited to, augmenting the visible light image with the data,adding data to the three dimensional colon model 170, and/or somecombination thereof.

Location data may be relative to a known reference point in space toallow the operator to locate the loop 184 or feature 180 (e.g., polyp orother object within the body of the patient). The reference point mayinclude, but is not limited to, the point of insertion into the body ofthe patient or any other reference point configurable to the operator.Knowing the exact location of the loop 184, for example, with respect tothe reference point in real time may allow the operator to immediatelyaffect treatment of the loop 184 and/or feature 180. Additionally, usingmethods similar to detection of features 180 and loop 184, the controlsystem 16 can identify the existence of other abnormal tissues, such asa tumor, alert the operator to the presence of the tissue, and provideexact location data to the operator as shown in FIG. 36F. For each, thecontrol system 16 can save location, size and related data in storedmemory for later use. Stored memory may include, but not limited to,memory within control system 16 and/or digital record repositoriesoutside of the control system 16 or colonoscopy system 10 f.

In some embodiments, the operator may navigate the endoscope 22 in sucha way that causes the operator to not observe some portion of the colon164. The continually updated (e.g., dynamic) three dimensional model 170may, as a result, contain only a partial image of a section of the colon164. The control system 16 may alert the operator that the opticalscanning system 12 f did not entirely interrogate the section as shownin FIG. 36G. The operator may be alerted by using methods that include,but are not limited to, changing the color of the partially scannedsection on the three dimensional colon model 170 and/or augmenting thevisible light system images with visual and audible warnings, forexample. The alert and the three dimensional colon model 170 may allowthe operator to determine the location of the unscanned tissue. Thethree dimensional model 170 and the visible light imagery provide visualguidance to the operator for maneuvering the endoscope 22 to allow thescanning system to fill in areas that were not originally scanned anddid not appear in the three dimensional colon model 170. The visualguidance may include, but is not limited to, the three dimensional modelincluding icons, arrows or other prompts overlaid on the visible lightimagery.

In some embodiments, the three dimensional model 170 may allow theoperator to visually verify and officially document that full cecalintubation—complete scanning of the colon 164 to the cecum—did occur, asshown in FIG. 36H. As the operator retracts the endoscope 22 from thepatient, the colonoscopy system 10 f can obtain another scan of eachsection of the colon 164. Using data related to registration, thecontrol system 16 may update data to and update data within the threedimensional colon model 170. As such, the control system 16 can refinethe three dimensional colon model 170, increasing model content andaccuracy, refining positions of polyps, adenomas, and tumors, and/orupdating any changes in the colon 164. An exemplary update may include,for example, but is not limited to, reimaging the colon 164 after thephysician removes the colon loop 184. FIG. 36I depicts the revised threedimensional model 170 of the colon 164, that when compared to FIG. 36Eillustrates removal of the colon loop 184.

In some embodiments, the algorithms may allow for the three dimensionalmodel 170 and all imagery augmentation to occur in real-time orsubstantially real time during the procedure as the optical scanningsystem 12 f traverses the colon 164. The operator may be provided withreal-time imagery, detection capability, and data in real time orsubstantially real time.

From the above description, it is clear that the inventive concept(s)disclosed herein are well adapted to carry out the objects and to attainthe advantages mentioned herein, as well as those inherent in theinventive concept(s) disclosed herein. While the embodiments of theinventive concept(s) disclosed herein have been described for purposesof this disclosure, it will be understood that numerous changes may bemade and readily suggested to those skilled in the art which areaccomplished within the scope and spirit of the inventive concept(s)disclosed herein.

The following references are referred to herein.

-   [1] M. Than, J. Witherspoon, J. Shami, P. Patil, A. Saklani,    “Diagnostic miss rate for colorectal cancer: an audit,” Annals of    Gastroenterology vol. 28, pp. 94-98, 2015-   [2] S. B. Ahn, D. S. Han, J. H. Bae, T. J. Byun, J. P. Kim, C. S.    Eun, “The miss rate for colorectal adenoma determined by    quality-adjusted, back-to-back colonoscopies,” Gut and Liver, vol.    6, no. 1, pp. 64-70, 2012.-   [3] E. B. Bressler, L. F. Paszat, Z. Chen, D. M. Rothwell, C.    Vinden, L. Rabeneck, “Rates of new or missed colorectal cancers    after colonoscopy and their risk factors: a population-based    analysis,” Gastroenterology, vol. 132, no. 1, pp. 96-102, 2007.-   [4] A. S. Xirasagar, T. G. Hurley, L. Sros, J. R. Hebert, “Quality    and safety of screening colonoscopies performed by primary care    physicians with standby specialist support,” Med Care, vol. 48, no.    8, pp. 703-709, August 2010.-   [5] E. S. Dellon, Q. K. Lippmann, R. S. Sandler, M D, and N. J.    Shaheen, “GI endoscopy nurse experience and polyp detection during    screening colonoscopy,” Clin Gastroenterol Hepatol. 6(12):    1342-1347. December 2008.-   [6] C. D. A. Joseph, R. G. S. Meester, A. G. Zauber, D. L.    Manninen, L. Winges, F. B. Dong, B. Peaker, M. van Ballegooijen,    “Colorectal cancer screening: estimated future colonoscopy need and    current volume and capacity,” Cancer, p. 2479-2486, August 2016.-   [7] American Cancer Society    https://www.cancer.org/cancer/colon-rectal-cancer/about/key-statistics.html-   [8] B. CDC Vital Signs, “Colorectal Cancer,” p. 1-3, July 2011.-   [9] NIH National Cancer Institute, Cancer Prevalence and Cost of    Care Projections,    https://costprojections.cancer.gov/expenditures.html-   [10] G. J. Pan, L. Xin, Y.-F. Ma, L.-H. Hu, Z.-S. Li, “Colonoscopy    reduces colorectal cancer incidence and mortality in patients with    non-malignant findings: a meta-analysis,” Am. J. Gastroenterol, vol.    111, pp. 355-365, 2016.-   [11] Y. Backes, M. P. Schwartz, F. Borg, F. Wolfhagen, J. Groen, W.    Nederveen Cappel, J. Bergeijk, J. Geesing, B. W. M. Spanier, P.    Didden, F. Vleggaar, M. M. Lacle, S. Elias, and L. M. G. moons,    “Multicentre prospective evaluation of real-time optical diagnosis    of T1 colorectal cancer in large non-pedunculated colorectal polyps    using narrow band imaging,” Endoscopy, BMJ Journals, Vol. 68, Issue    2, 2019.-   [12] P. Lefere, S. Gryspeerdt, “Virtual colonoscopy: A practical    guide,” Springer, 2009.-   [13] M. Rubin, L. Lurie, K. Bose, S. H. Kim, “Expanding the view of    a standard colonoscope with the third eye panoramic cap,” World    Journal of Gastroenterology, Vo. 21(37), pp. 10683-10687, 2015.-   [14] ASGE Technology Committee, “Endoscopes and devices to improve    colon polyp detection,” Technology Status Evaluation Report,    Gastrointestinal Edoscopy, vol. 81, no. 5, pp. 1122-1129, 2015.-   [15] https://endocuff.com/-   [16]    http://www.smartmedsys.com/index.php/products/naviaid-g-eye/overview-   [17] http://www.usendoscopy.com/products/endorings-device-   [18] Avanti Medical Systems, https://www.avantismedicalsystems.com/-   [19] Shin, Younghak, et al. “Automatic colon polyp detection using    region based deep cnn and post learning approaches.” IEEE Access 6    (2018): 40950-40962.-   [20] Wang, Pu, et al. “Development and validation of a deep-learning    algorithm for the detection of polyps during colonoscopy.” Nature    biomedical engineering 2.10 (2018): 741.-   [21] J. W. Leung, A. Thai, A. Yen, G. Ward, O. Abramyan, J Lee, B.    Smith, F. Leung, “Magnetic endoscope imaging (ScopeGuide) elucidates    the mechanism of action of the pain-alleviating impact of water    exchange colonospoy—attenuation of loop formation,” J Intery    Gastroenterol, Vol 2:3, pp. 142-146, 2012.-   [22] Olympus (ScopeGuide),    http://medical.olympusamerica.com/technology/scopeguide-   [23] Global Market Insights, Endoscopy Market,    https://www.gminsights.com/pressrelease/endoscopy-market-   [24] Endoscopy Equipment Market by Product and Application Global    Forecast, https://www.marketsandm    arkets.com/Market-Reports/endoscopy-devices-market-689.html?gclid=CjwKCAiAgrfhBRA3EiwAnfF4tviHObO-MEy-q7SeXyr6FdZrHWBvG08MWCkAadqlxIQNXePADqy2NBoCPKgQAvD_BwE-   [25] Olympus, Gastroenterology    https://medical.olympusamerica.com/specialty/gastroenterology-   [26] Boston Scientific, Gastroenterology    http://www.bostonscientific.com/en-US/medical-specialties/gastroenterology.html-   [27] Auris health, https://www.aurishealth.com/home.html-   [28] P. Zanuttigh, G. Marin, C. D. Mutto, F. Dominio, L.    Minto, G. M. Cortelazzo, “Time-of-flight and structured light depth    cameras,” Chapter 3, Springer 2016.-   [29] G. Frankowski, R. Hainich, “DLP-Based 3D metrology by    structured light or projected fringe technology for life sciences    and industrial metrology,” Proceedings of the SPIE, vol. 7210, pp.    72100C-1 to 72100C-12, 2009.-   [30] G. Frankowski, R. Hainich, “DLP/DSP-based optical 3D sensors    for the mass market in industrial metrology and life sciences,”    Proceedings of the SPIE, Photonics West, 2011.-   [31] J. Geng, “Structured-light 3D surface imaging: a tutorial,”    Advances in Optics and Photonics, vol. 3, pp. 128-160, 2011.-   [32] K. Harding, ed., “Handbook of Optical Dimensional Metrology,”    CRC Press, Boca Raton, Fla., Chapters 3-5, 2013.-   [33] H. Nguyen, D. Nguyen, Z. Wang, H. Kieu, M. Le, “Real-time,    high-accuracy 3D imaging and shape measurement,” Applied Optics,    vol. 54, no. 1, pp. A9-A17, 2015.-   [34] Joseph Daniel Bokusky, and Enrique Romo, “Methods and apparatus    for constructing endoscopic device with helical lumen design,” Auris    Surgical Robotics, U.S. Pat. No. 9,844,412, Dec. 19, 2017.-   [35] Alan Yu, and Jason Lee, “Articulating flexible endoscopic tool    with roll capabilities,” Auris Surgical Robotics, U.S. Pat. No.    9,561,083, Feb. 7, 2017.-   [36] Mark Harris, “First Surgical Robot from Secretive Startup Auris    Cleared for Use,” IEEE Spectrum, Jun. 7, 2016.-   [37] Alan, Yu, Frederic H. Moll, Benjamin Richter, Mark H. Olson,    Jason Gonzalez, Kyle Andrew Tucker, Paxton Maeder-York, and Gregory    Schulte, “Surgical robotics system,” Auris Surgical Robotics, U.S.    Pat. No. 9,622,827, Apr. 18, 2017.-   [38] David Mintz, Atiyeh Ghoreyshi, Prasanth Jeevan, Yiliang Xu,    Gehua Yang, Mathew Joseph Leotta, Charles Stewart, “Navigation of    tubular networks,” Auris Surgical Robotics, U.S. Pat. No. 9,727,963,    Aug. 8, 2017.-   [39] Jeffery Alvarez, Jian Zhang and Alisha Seam, “Method, apparatus    and a system for robotics assisted surgery,” Auris Surgical    Robotics, US 2014/0142591, May 22, 2014.-   [40] Jason Lee, Christopher Sramek, Gregory Kintz, David Mintz, and    Alan Yu, “Floating electromagnetic field generator system and method    of controlling the same,” Auris Surgical Robotics, US 2017/0290631,    Oct. 12, 2017.-   [41] Travis Schuh, Mathew Reagan Williams, Joseph Daniel Bogusky,    David Mintz, Alan Yu, and Yoichiro Dan, “Instrument device    manipulator with back-mounted tool attachment mechanism,” Auris    Surgical Robotics, US 2017/0367782, Dec. 28, 2017.-   [42] Travis Schuh, “Instrument-mounted tension sending mechanism for    robotically-driven medical instruments,” Auris Surgical Robotics,    U.S. Pat. No. 9,788,910, Oct. 17, 2017.-   [43] Travis Schuh, “Instrument device manipulator and surgical    drape,” Auris Surgical Robotics, U.S. Pat. No. 9,737,373, Aug. 22,    2017.

What is claimed is:
 1. A system, comprising: at least one opticalscanning system configured to scan and capture at least one image of athree-dimensional environment within a colon, the optical scanningsystem comprising: at least one illuminator configured to producespatially patterned light and solid light in at least one frame toilluminate tissue within the colon; at least one camera configured tocapture the at least one image of the illuminated tissue within thecolon; at least one control system configured to construct at least onethree dimensional point cloud representations of the tissue within thecolon.
 2. The system of claim 1, wherein the at least one opticalscanning system is within a housing mounted to an endoscopic hardware.3. The system of claim 1, wherein the at least one optical scanningsystem is integrated within an endoscopic hardware.
 4. The system ofclaim 1, wherein the illuminator includes at least one VCSEL array, atleast one projection optic and at least one diffractive optical element.5. The system of claim 1, wherein the at least one illuminator isconfigured to produce spatially patterned light and solid light in atleast one separate frame.
 6. The system of claim 1, wherein the at leastone illuminator is configured to produce spatially patterned light andsolid light in at least one composite frame.
 7. The system of claim 1,wherein operating wavelength of the illuminator and the camera includesinfrared wavelengths.
 8. The system of claim 1, wherein operatingwavelength of the illuminator and the camera includes visiblewavelengths.
 9. The system of claim 1, wherein the illuminator andcamera are configured geometrically and optically to be used with atleast one triangulation algorithm.
 10. The system of claim 1, whereinthe control system uses at least one triangulation algorithm to processdata from the camera and illuminator to construct the one or more threedimensional point cloud representations of the tissue within the colon.11. The system of claim 1, wherein the control system uses data from atleast two images captured by at least one camera to determine rotationand translation of the optical scanning system between frames.
 12. Thesystem of claim 11, wherein the control system uses rotation andtranslation of the optical scanning system and the at least one threedimensional point cloud representation to provide at least one threedimensional model of the tissue within the colon.
 13. The system ofclaim 12, wherein the three dimensional model traverses an extendedlength of the colon.
 14. The system of claim 13, wherein the extendedlength of the colon includes an entire length traversed by the opticalscanning system through the colon.
 15. The system of claim 1, whereinthe at least one control system detects presence of a feature within thecolon selected from a group consisting of a polyp or an adenoma usingthe at least one three dimensional point cloud.
 16. The system of claim15, wherein the at least one control system measures the feature withinthe colon in a lateral and depth dimension.
 17. The system of claim 15,wherein the control system provides data to an endoscopic system toguide positioning of at least one instrument for removal of the feature.18. The system of claim 1, wherein the control system uses data of thethree dimensional point cloud, mapping and registration data to detectloops in the colon.
 19. The system of claim 1, wherein the controlsystem uses data of the three dimensional point cloud, mapping andregistration data to collect baseline data for a patient.
 20. The systemof claim 1, wherein the control system uses data of the threedimensional point cloud, mapping and registration data to mark locationof tissue requiring further analysis.
 21. The system of claim 1, whereinthe control system further analyzes data of the three dimensional pointcloud, mapping and registration data with real time data provided byoperation of the optical scanning system within the colon.
 22. Thesystem of claim 21, wherein the control system provides at least onealert to an operator of the optical scanning system.
 23. The system ofclaim 1, wherein the optical scanning system includes at least oneproximity sensor.